Robots.txt Generator
Create a custom robots.txt with AI crawler controls.
Want full SEO blog posts written and published for you?
SEOGraphy generates complete articles with images, then publishes them directly to WordPress — after you review and approve.
What does robots.txt do?
robots.txt is a plain text file placed at the root of your website that tells web crawlers which pages and folders they can and cannot access. Every major search engine bot (Googlebot, Bingbot) and AI crawler (GPTBot, ClaudeBot, PerplexityBot) checks this file before crawling your site.
Blocking AI crawlers: the 2026 decision
Many publishers are now blocking AI training crawlers (GPTBot, CCBot, etc.) to prevent their content from being used to train AI models without compensation. However, this is separate from AEO/GEO optimisation — you can block training crawlers while still being cited by AI search tools (which use different crawler identifiers). This generator includes the most common AI crawler identifiers for 2026.
Want full articles written for you?
SEOGraphy writes SEO-optimised articles, generates images, and publishes to WordPress — automatically.
Start Free — No Credit Card