Aveena
Free tool

Robots.txt Generator

Generate a correct robots.txt with rules for search engines and AI crawlers - decide exactly which AI bots can read and cite your site.

robots.txt and AI crawlers

robots.txt is the file at your site root that tells crawlers which URLs they may visit. AI companies now run their own named crawlers, so the same file is where you decide whether ChatGPT, Claude, Perplexity, Gemini and the rest can read your content - and therefore whether they can cite you at all.

Used for the Sitemap directive.

OAI-SearchBot

OpenAI · ChatGPT search

ChatGPT-User

OpenAI · user browsing

GPTBot

OpenAI · model training

ClaudeBot

Anthropic · Claude

PerplexityBot

Perplexity

Google-Extended

Google · Gemini training

Amazonbot

Amazon · Alexa/Rufus

Applebot-Extended

Apple · AI training

CCBot

Common Crawl · training

Bytespider

ByteDance · training

meta-externalagent

Meta · AI training

One path per line (applies to all crawlers).

# robots.txt - generated by Aveena (aveena.co.uk)

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt at the root of your site (/robots.txt).

Should you block AI crawlers?

There is a genuine trade-off here, and anyone who tells you the answer is obvious is selling something.

The case for blocking: your content is used to train commercial models you get nothing back from. If your business is the content itself - publishing, research, courses, anything paywalled - training crawlers take the value and return neither traffic nor payment. Some also crawl hard enough to cost you real server resources.

The case for allowing: you cannot be cited by something that cannot read you. As people move from searching to asking, an AI answer that mentions your brand is the new front page - and blocking the crawlers behind it makes you invisible there. For most businesses, being absent from AI answers costs more than the training data is worth.

The middle ground is usually right: allow the crawlers that produce citations and send traffic, block the ones that only feed training. That distinction is what the presets below are built around.

GPTBot (OpenAI)

GPTBot is OpenAI’s crawler for model training. Blocking it opts your content out of future training runs. The important nuance: it is not the only OpenAI agent. OAI-SearchBotbuilds the index behind ChatGPT’s search, and ChatGPT-User fetches a page when someone asks about it directly. Block GPTBot and you stay out of training but remain citable in ChatGPT - block all three and you disappear from it entirely.

ClaudeBot (Anthropic)

ClaudeBot is Anthropic’s crawler for Claude. Claude cites sources when it answers from the web, so allowing ClaudeBot is what makes your pages available to be surfaced and linked. Blocking it removes you from that pool. For most businesses this is a citation crawler rather than a threat - if you want to be mentioned in Claude’s answers, it needs to be able to read you.

PerplexityBot (Perplexity)

PerplexityBot crawls for Perplexity, which is the most citation-heavy of the answer engines - it footnotes almost every claim with a link. That makes it one of the few AI engines that reliably sends referral traffic back to you. Blocking PerplexityBot is usually a straightforward loss: you give up the citations and the clicks that come with them.

Google-Extended (Google)

Google-Extended isn’t a crawler at all - it’s a control token you put in robots.txt to say whether your content may be used to train and ground Google’s generative models, such as Gemini. Googlebot still crawls your site for Search either way, and Google states that using Google-Extended does not affect your Search rankings. It lets you opt out of the AI training use without touching your search visibility.

CCBot (Common Crawl)

CCBot belongs to Common Crawl, a non-profit that publishes an open web archive. It isn’t an AI company, but its dataset is one of the most widely used training corpora in the industry, so blocking CCBot removes your content from a source many different models draw on at once. It’s the highest-leverage single block if your goal is to stay out of training data.

Bytespider (ByteDance)

Bytespider is ByteDance’s crawler, gathering data for its AI products. It has a reputation for crawling aggressively, so it is often blocked for server load as much as for principle. There is little citation upside in return, which makes it one of the easier decisions on this list for most sites.

Applebot-Extended (Apple)

Applebot-Extended is Apple’s equivalent of Google-Extended: an opt-out token for using your content to train Apple Intelligence. Plain Applebotis separate and still powers Siri and Spotlight search results. Blocking Applebot-Extended keeps you out of the AI training while leaving Apple’s search features working as normal.

Presets built into the generator

Rather than making you reason about eleven user-agents one at a time, the generator ships three starting points:

  • Maximise AI visibility - allow every AI crawler. The right default if you want to be found and cited.
  • Allow answers, block training - let the assistants that cite you read your site, while blocking training-only agents. The balanced option.
  • Block all AI crawlers - opt out entirely, for sites whose content is the product.

Pick a preset, adjust any individual bot, add your disallowed paths and sitemap, then copy or download the file. Pair it with an llms.txt file so the crawlers you do allow get a clean map of your content, and with a strong entity footprint so they know who you are once they arrive.

Related

Frequently asked questions

Will blocking AI crawlers hurt my Google rankings?

No. Googlebot handles Search and is controlled separately from Google-Extended, which only governs generative AI training and grounding. Blocking the AI-specific agents doesn't change how Google crawls or ranks your pages for normal search results.

Does robots.txt actually stop AI crawlers?

It's a convention, not a technical barrier. The major, named crawlers from OpenAI, Anthropic, Google, Apple and Perplexity do respect robots.txt. Less reputable scrapers may ignore it entirely, so treat robots.txt as a clear statement of intent rather than enforcement.

Where does robots.txt need to live?

At the root of your domain, so it resolves at yourdomain.com/robots.txt. It only applies to the host it's served from, so a subdomain needs its own file. It's also the right place to reference your sitemap URL.

Can I allow AI answers but block AI training?

Yes, and for most businesses it's the sensible middle ground. Allow the citation crawlers that surface and link to you - OAI-SearchBot, ClaudeBot, PerplexityBot - while blocking the training-only agents like GPTBot, CCBot, Google-Extended and Applebot-Extended. The generator has a preset that does exactly this.