Create a custom robots.txt with AI crawler controls.
Want full SEO blog posts written and published for you?
SEOGraphy generates complete articles with images, then publishes them directly to WordPress - after you review and approve.
Join the waitlist and we will email you a personal discount code that stacks an extra 10% on top of launch pricing the day SEOGraphy goes live. The free tools stay free either way.
robots.txt is a plain text file placed at the root of your website that tells web crawlers which pages and folders they can and cannot access. Every major search engine bot (Googlebot, Bingbot) and AI crawler (GPTBot, ClaudeBot, PerplexityBot) checks this file before crawling your site.
Many publishers are now blocking AI training crawlers (GPTBot, CCBot, etc.) to prevent their content from being used to train AI models without compensation. However, this is separate from AEO/GEO optimisation - you can block training crawlers while still being cited by AI search tools (which use different crawler identifiers). This generator includes the most common AI crawler identifiers for 2026.
Want full articles written for you?
SEOGraphy writes SEO-optimised articles, generates images, and publishes to WordPress - automatically.
Start Free - No Credit Card