Fuck AI

If you are using Hugo use this robots.txt template that automatically updates every build:

{{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
{{- $resource := resources.GetRemote $url -}}
{{- with try $resource -}}
  {{ with .Err }}
    {{ errorf "%s" . }}
  {{ else with .Value }}
	{{- .Content -}}
  {{ else }}
    {{ errorf "Unable to get remote resource %q" $url }}
  {{ end }}
{{ end -}}

Sitemap: {{ "sitemap.xml" | absURL }}

Optionally lead rouge bots to poisoned pages:

{{- $url := "https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/refs/heads/main/robots.txt" -}}
{{- $resource := resources.GetRemote $url -}}
{{- with try $resource -}}
  {{ with .Err }}
    {{ errorf "%s" . }}
  {{ else with .Value }}
    {{- printf "%s\n%s\n\n" "User-Agent: *" "Disallow: /train-me" }}
    {{- .Content -}}
  {{ else }}
    {{ errorf "Unable to get remote resource %q" $url }}
  {{ end }}
{{ end -}}

Sitemap: {{ "sitemap.xml" | absURL }}

~~Check out how to poison your pages for rouge bots in this article~~

Repo was deleted and the internet archive was excluded.

I use Quixotic and a Python script to poison the pages and I included those in my site update script.

Its all cobbled together in amateur fashion from the deleted article but its honest work.

That's pretty sweet but just be aware a lot of bots are bad actors and don't advertise a proper user agent, so you have to also block by ips. Blocking all alibaba server ips is a good start.

Most AI crawlers don't respect robots.txt files, but this info might be useful for other forms of blocking.

Many AI crawlers don't identify properly, they fake the User Agent and pretend to be Google Chrome or something like that. So it's bound to only deal with the ones that somehow behave, while it won't do anything to the real bad ones. And from my experience, those can do enough requests per second to get an average server into trouble. At least that's what happened to mine.

If only they could read