robots.txt for AI agents
2025-03-24 03:26:34.850469+01 by Dan Lyke 0 comments
I really need to write or install a bot fucker-upper... AI6YR Ben @ai6yr@m.ai6yr.org
List of AI bots to add to robots.txt (although they may not obey -- may need to throw them in the bitbucket and 404 or 444 them). In addition to these, you may have to block specific random browser versions for the most aggressive bots who ignore robots.txt.
https://github.com/ai-robots-t....robots.txt/blob/main/robots.txt