Why aren’t you de-duping the table before processing?
I need to dedupe the to-be-processed data with the data thats already processed in the "final" table. We are working with hundreds of millions of products therefore we thought about "simply" using random batches from the data to be processed. But thanks to the many replies Ive learned already that our approach was in the beginning already wrong.
es is schowieder jeder krank. 🥲 i dafias überhaupt net.
wennst im bus sitzt kommen influenza vibes hoch.
ich leid mit dir. gute besserung.
des geht si dann nur wieder mit am spesenkonto aus.
yes. the problem is, we are fetching products from an API. and since we thought processing power will be a limiting factor, we thought that sorting out duplicates would reduce load.
but since the different microservices which process the data are taking different times we are using the sql tables as a pool. this should help upcscaling by using multiple microservices.
cloud services are yet not a solution as we are still in development.
haha. relating to a switch to ssd? or in which direction?
wenn nicht jetzt. wann dann. klatsch klatsch
thats true. when I tried it I even found it more difficult because of the limite. you must trust the scripts and if something does not work its a lot more complicated than setting it up by yourself.
what do you meant with burnt? i thought its even easier than yunohost?
you rolling up an ai of?
wow. never thought of this in that way.
mir auch vor kurzem passiert. sql is a hund.
thanks for this input. This was the winning argument for my boss for migrating to a modern server. While I admit that I see many flaws in our design, we are now working on refactoring our architecture and approach itself.
Thanks to the other numerous answers leading me to the right direction (hopefully).