4.1k stars · MIT · v1.52 (2026-09-07) · Track this in Scout
A maintained list of AI crawler user-agents with ready-made robots.txt, nginx, Caddy and HAProxy rules.
▶Repo detailsthe review · specs · pros & cons · install
What it is
A community-maintained list of AI crawler user-agents, published as ready-made configuration files for several web servers: robots.txt, an Apache .htaccess, an Nginx include, a Caddyfile, an HAProxy list and a Lighttpd include. The robots.txt file asks politely. The server files refuse the request outright.What it is good for. Anyone publishing writing they would rather not have harvested, which is most people who write anything. The problem it removes is not the blocking, which is easy, but the keeping up, which is not. Version 1.52 was published on 7 September 2026 and adds several newly spotted robots, which is the whole value: somebody else is watching. The Repo Radar archive at grasppy.com/radar is exactly the sort of page this is for — long, original, and useful to a scraper.
- Five minutes, one file, nothing running afterwards, and no cost of any kind.
- The list is updated constantly and tagged as a release, so you can subscribe to the releases feed and know when to update.
- It ships configuration for six different web servers, so you almost certainly do not have to translate anything.
- A
robots.txtfile is a request, not a wall. Well-behaved crawlers obey it and badly behaved ones ignore it. The web server files are what actually refuse them. - Blocking by user-agent name is easy to get around. A scraper that lies about its name walks straight past this.
- Blocking too widely can hide you from search engines or AI assistants that send you readers. Read the list before you apply it, and decide which ones you actually want kept out.
- TecharoHQ/anubis
Takes a different approach entirely: it sits in front of your site and makes every visitor's browser do a small piece of work first, which costs a scraper running millions of requests far more than it costs a reader.
Track this in Scout - mitchellkrogza/nginx-ultimate-bad-bot-blocker
A much larger Nginx blocklist covering spam referrers, scanners and fake search-engine robots as well as AI ones, so it is broader and noisier.
Track this in Scout - google/robotstxt
Google's own parser for robots.txt files, released as a library and a test tool, useful for checking that the rules you wrote say what you think they say.
Track this in Scout
# Nginx. Fetch the include file and point your server block at it:
sudo curl -o /etc/nginx/nginx-block-ai-bots.conf \
https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/nginx-block-ai-bots.conf
# Then inside your server { } block add:
# include /etc/nginx/nginx-block-ai-bots.conf;
sudo nginx -t && sudo systemctl reload nginx
# The polite half, for any web server. Put this at the site root:
curl -o robots.txt \
https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/robots.txt
# To be told when the list changes, subscribe to:
# https://github.com/ai-robots-txt/ai.robots.txt/releases.atom