robots.txt Generator
Control which AI and search crawlers can access your site — in seconds.
Allow the bots you want, block the ones you don't. No signup required.
What is robots.txt?
robots.txt is a file at your website's root that tells web crawlers which parts of your site they're allowed to access.
AI platforms like ChatGPT, Perplexity, Claude, and Gemini each have their own crawlers — separate from Google and Bing. If you block a crawler, that AI platform literally cannot see your content and will never recommend you.
Most businesses should allow all AI crawlers unless they have a specific reason to block one. Use the toggles below to customize your file.
AI Crawlers
Toggle each bot to allow or block it from crawling your site.
Search Crawlers
Most sites should leave these allowed.
Additional settings
Live preview
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: cohere-ai Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: * Allow: /
robots.txt in your website's root so it's served at yoursite.com/robots.txt.Allowing AI bots isn't enough — they also need rendered HTML
Most AI crawlers don't run JavaScript. RenderBeam delivers fully-rendered pages so AI platforms can actually read your content. Free for 30 days.
Try RenderBeam FreeGo further with RenderBeam
Keep going — here are a few good next stops.