384
Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries
(arstechnica.com)
This is a most excellent place for technology news and articles.
They can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.