this post was submitted on 26 Mar 2025
384 points (99.0% liked)

Technology

68187 readers
5310 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
384
Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries (arstechnica.com)
submitted 1 week ago by cm0002@lemmy.world to c/technology@lemmy.world
25 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] mesamunefire@lemmy.world 83 points 1 week ago (3 children)

Yep same thing. I have some small servers and was getting hammered by openai ip controlled ai crawlers not respecting robots.txt. had to block all their IP addresses and create an AI black hole in order to stop them ddos ing my tiny site(s).

[–] Tangent5280@lemmy.world 2 points 5 days ago (1 children)

Hey, could you say how you did that? I'm looking to put a few servers up and I'm worried about this too

[–] mesamunefire@lemmy.world 3 points 5 days ago* (last edited 5 days ago) (1 children)

I used fail2ban + router to block the ip addresses. Then if the headers come from openai, they also get bounced.

Below is a template I used that I created on the fly for an AI black hole that I also made. Its decent, but I feel like it could be better.

from flask import Flask, request, redirect, render_template_string
import time
from collections import defaultdict
import random

app = Flask(__name__)

# Data structure to keep track of requests per IP
ip_requests = defaultdict(list)
IP_REQUEST_THRESHOLD = 1000  # Requests threshold for one hour
TIME_WINDOW = 3600  # Time window of one hour in seconds

# Function to track and limit requests based on IP
def track_requests(ip):
    current_time = time.time()
    ip_requests[ip] = [t for t in ip_requests[ip] if current_time - t < TIME_WINDOW]  # Remove old requests
    ip_requests[ip].append(current_time)
    return len(ip_requests[ip])

# Serve slow pages incrementally
@app.route('/')
def index():
    ip = request.remote_addr
    request_count = track_requests(ip)

    if request_count > IP_REQUEST_THRESHOLD:
        return serve_slow_page(request_count)
    else:
        return 'Welcome to the site!'

def serve_slow_page(request_count):
    """Serve a progressively slower page."""
    delay = min(10, request_count / 1000)  # Slow down incrementally, max 10 seconds delay
    time.sleep(delay)  # Delay to slow down the request

    # Generate the next "black hole" link
    next_page_link = f'/slow/{random.randint(1000, 9999)}'
    
    html_content = f"""
    <html>
    <head><title>Slowing You Down...</title></head>
    <body>
        <h1>You are being slowed down!</h1>
        <p>This is taking longer than usual because you're making too many requests.</p>
        <p>You have made more than {IP_REQUEST_THRESHOLD} requests in the past hour.</p>
        <p>Next step: <a href="{next_page_link}">Click here for the next page...</a></p>
    </body>
    </html>
    """
    return render_template_string(html_content)

@app.route('/slow/<int:page_id>')
def slow_page(page_id):
    ip = request.remote_addr
    request_count = track_requests(ip)

    if request_count > IP_REQUEST_THRESHOLD:
        return serve_slow_page(request_count)
    else:
        return 'Welcome back to normal!'

if __name__ == '__main__':
    app.run(debug=True)
[–] Tangent5280@lemmy.world 2 points 5 days ago

Thanks, this will help.

[–] tempest@lemmy.ca 48 points 1 week ago

Did you not pay your protection money to CloudFlare?

[–] CosmicCleric@lemmy.world 27 points 1 week ago (1 children)

Man, this current age of AI really sucks.

~This~ ~comment~ ~is~ ~licensed~ ~under~ ~CC~ ~BY-NC-SA~ ~4.0~

[–] Burghler@sh.itjust.works 16 points 1 week ago (2 children)

You licensed your comment? You just saw first hand that AI crawlers don't care about legal barriers.

[–] WhyJiffie@sh.itjust.works 6 points 1 week ago

I suppose the goal isa kind of poisoning the well, in case at any time in the future it becomes enforceable somehow