There is a file at the root of your site, probably four lines long, that most owners have never read. It is currently deciding whether ChatGPT, Claude, Perplexity, and Google's AI surfaces are permitted to fetch your pages.
Plenty of platforms and security plugins block AI crawlers by default. If nobody made a deliberate decision, a default made it.
Who Is Actually Knocking
GPTBot, OpenAI's crawler for training and improving models. OAI-SearchBot, OpenAI's crawler for its search and browsing features, which is the one that matters most for live retrieval.
ClaudeBot, Anthropic's crawler. PerplexityBot, Perplexity's crawler, feeding a product that is explicitly a citation engine.
Google-Extended, which is not Googlebot. This is a separate control governing whether your content can be used across Google's generative AI surfaces. Blocking it does not remove you from search results. It does remove you from an increasing share of the answers Google generates above them.
Applebot-Extended, the equivalent control for Apple's AI features. Bingbot, which is upstream of several assistants and worth allowing on its own merits.
The Decision, Made Honestly
The tradeoff is not complicated once it is stated plainly.
Allowing means your pages can be retrieved, summarized, and cited when someone asks an assistant who to hire. It also means your content may be used in training, which for a service business is not a loss. Being encoded as a known entity in a model is the outcome you want.
Blocking means compliant crawlers stop fetching. It does not protect anything from bad actors, who never read the file. And it removes your business from consideration in a channel where buyers are increasingly making shortlist decisions.
For a publisher whose product is the article itself, blocking can be a coherent business position. For a service business trying to be recommended, blocking is opting out. The mechanics of how those recommendations get made are here.
The Sensible Configuration
- Allow the retrieval and indexing crawlers explicitly. Name them rather than relying on a permissive wildcard, so the intent is unmistakable and survives the next platform update.
- Disallow only what should genuinely never be public. Admin paths, checkout flows, internal search result pages, staging directories, thank-you pages.
- Reference the sitemap. A full sitemap URL in robots.txt is the cheapest crawl-efficiency win available.
- Do not block CSS and JS. Blocking them prevents rendering-capable crawlers from seeing the page as a user does, which is a self-inflicted wound.
- Verify the file is actually served. Fetch it. Plenty of sites have a robots.txt in a repository that never reaches the domain root.
The File Most Sites Are Missing
robots.txt grants permission. It does not explain anything.
An llms.txt file at the site root does. It is plain markdown stating what the business is, what it sells, what it costs, who it serves, where it operates, and what questions it answers, written for a language model rather than for a person.
It is not an official standard and no engine guarantees it will read it. It costs an hour, it is trivially maintainable, and when a model does fetch it, it gets a clean, unambiguous, first-party summary instead of inferring the business from a marketing page. That asymmetry is why it is worth shipping.
The thing that makes it work is the same thing that makes everything else in this layer work: the facts in it must match the facts on the site, in the schema, and on every off-site profile. Consistency is the currency.
UM Media configures robots, sitemap, schema, and llms.txt on every build and in every SEO / AEO / GEO Foundation. See what the Foundation covers, or the connected Brand System.