Create a custom robots.txt with AI crawler controls.
Want full SEO blog posts written and published for you?
SEOGraphy generates complete articles with images, then publishes them directly to WordPress - after you review and approve.
SEOGraphy is live. Subscribe before 31 October and keep 50% off for the lifetime of your plan - the discount never expires, the offer does.
robots.txt is a plain text file placed at the root of your website that tells web crawlers which pages and folders they can and cannot access. Every major search engine bot (Googlebot, Bingbot) and AI crawler (GPTBot, ClaudeBot, PerplexityBot) checks this file before crawling your site.
Many publishers are now blocking AI training crawlers (GPTBot, CCBot, etc.) to prevent their content from being used to train AI models without compensation. However, this is separate from AEO/GEO optimisation - you can block training crawlers while still being cited by AI search tools (which use different crawler identifiers). This generator includes the most common AI crawler identifiers for 2026.
Want full articles written for you?
SEOGraphy writes SEO-optimised articles, generates images, and publishes to WordPress - automatically.
Start Free - No Credit Card