The only simple possibles ways are:

• robot.txt
• rate limiting by ip
• blocking by user agent

From the article, they try to bypass all of them:

They also don't give a single flying fuck about robots.txt ...

If you try to rate-limit them, they'll just switch to other IPs all the time. If you try to block them by User Agent string, they'll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

It then become a game of whac a mole with big tech 😓

~~The more infuriating for me is that it's done by the big names, and not some random startup.~~ Edit: Now that I think about it, this doesn't prove it is done by Google or Amazon: it can be someone using random popular user agents