---
title: "Postmortem: The Bot Block That Kept a Business Out of AI Answers"
description: "A composite postmortem of how an AI-bot setting can block live answer fetchers, why content work misses the problem, and how to check the edge, robots.txt, and rendered page."
url: "https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d773e603-2e35-4109-bf59-8eb3bd9fa4a5.png"
published: "2026-10-16T05:58:05.414Z"
modified: "2026-10-16T05:58:05.504Z"
---

October 16, 2026 · 10 min read

# Postmortem: The Bot Block That Kept a Business Out of AI Answers

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tabletop model office piles up stacks of pages while a tiny courier is turned away at its locked gate and walks to an open competitor shop next door.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d773e603-2e35-4109-bf59-8eb3bd9fa4a5.png)

In this article

1. [Setup: a site that looks healthy from inside](#setup-a-site-that-looks-healthy-from-inside)
2. [First sign: the answer names everyone else](#first-sign-the-answer-names-everyone-else)
3. [Wrong call: buy content before checking the door](#wrong-call-buy-content-before-checking-the-door)
4. [Damage: a full calendar, an empty path to the answer](#damage-a-full-calendar-an-empty-path-to-the-answer)
5. [Fix: separate training policy from answer access](#fix-separate-training-policy-from-answer-access)

A company can publish the right answer and still never get cited. If its CDN, firewall, or `robots.txt` blocks the bot fetching that answer, another site gets the chance instead.

I keep seeing the wrong diagnosis proposed for this problem: write more comparison pages, build more authority, buy a tracker. Those may be useful when an assistant can read the site. They do little when the request dies before the page loads. Over one 30-day window on groas infrastructure, `/robots.txt` received **1,222 verified requests from `Claude-User`**, Anthropic’s live-answer fetcher. That is a lot of door-checking to dismiss as background noise.

The case below is **a typical composite, not a claim about one identified company**. Its sequence is familiar: a missing citation, a content-first response, months of work, then the discovery that an anti-AI setting intended to discourage training crawlers also obstructed live retrieval. The distinction matters because the fix starts at the door, not in the editorial calendar.

## Setup: a site that looks healthy from inside

An executive asks ChatGPT for the best platforms for mid-market fleet dispatching. Two competitors appear, with links to pricing and case studies. Their company does not. The screenshot lands in Slack with the obvious question: _Why aren’t we listed?_

The marketing team checks the tools it knows. Google Search Console shows stable organic traffic. The site loads in a browser. Its comparison pages are current, and the content calendar is full. Nothing in that view says a live-answer fetcher might receive a different response at the edge.

That is the first trap. **A healthy Google report is not a test of whether an assistant can fetch a page.** A site can be available to a human visitor and still challenge or refuse an automated request. The assistant does not send the marketing team a ticket explaining what happened. The brand simply fails to appear in the answer.

## First sign: the answer names everyone else

The missing mention looks like a content problem. I understand the impulse. When a competitor gets cited, the instinct is to inspect its pages, rewrite yours, and publish more of them. But before spending the next quarter on that plan, I want to know whether the assistant could retrieve what the company has already published.

That requires separating three jobs commonly collapsed into the phrase _AI bot_. As [OpenAI’s bot documentation](https://developers.openai.com/docs/bots) illustrates, bots that gather material for model training are not the same as bots involved in search or user-requested page visits:

- **Training crawlers**, such as `GPTBot` and `ClaudeBot`, collect material for model development. A business may choose to disallow them.
- **Search crawlers**, such as `OAI-SearchBot`, `Claude-SearchBot`, and `PerplexityBot`, help make pages available for search-based answers.
- **User-triggered fetchers**, such as `ChatGPT-User`, `Claude-User`, and `Perplexity-User`, visit pages in connection with a user’s request.

![Diagram contrasting bulk AI training crawlers with fetchers seeking pages for live answers.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/cb5ea0f8-da11-4836-aa14-1d47aed0dd07.png)

A policy written as “block AI bots” can hit all three. That is how a reasonable decision about training data becomes a bad decision about being reachable when a buyer asks for current information. **The bot’s function matters more than its label.**

The groas log count makes the point concrete, with an important limit. Those 1,222 verified `Claude-User` requests show repeated checks of `/robots.txt`; they do not tell us what each person asked or prove that each check would have produced a citation. They do show why a rule at that endpoint deserves attention. [DataDome’s account of `Claude-User`](https://datadome.co/bot-management-protection/claude-user-anthropic/) describes its use of `robots.txt` during conversational retrieval. If access is refused there, better prose on the target page cannot help that fetch.

## Wrong call: buy content before checking the door

In the composite case, the agency treats the absent mention as an authority gap. It proposes twenty comparison articles, more third-party coverage, and an AI share-of-voice dashboard. The team approves the work. Writers write; product managers update feature tables; reports show that publishing is on schedule.

Then the same kind of prompt still leaves the company out.

I do not object to comparison content. I object to paying for it before checking delivery. A tracker can show that a brand was not mentioned in an answer. It cannot, from that answer alone, tell the team whether a CDN returned a `403 Forbidden`, an edge rule altered `robots.txt`, or the page arrived without its useful content. **“Not mentioned” is an outcome, not a diagnosis.**

Here is how the technical failure develops while the content work continues.

### 1. The anti-AI setting goes on

The original decision usually sits outside the marketing workflow. Someone reviews bot traffic or origin load and enables a protective setting. [Cloudflare’s AI-bot blocking announcement](https://blog.cloudflare.com/cloudflare-block-ai-bots/) made a one-click control available across plan levels. From an infrastructure perspective, blocking unwanted scraping can look like routine housekeeping. The trouble starts when nobody checks which agents the rule affects.

![Miniature server rack behind a security barrier that blocks incoming robotic mail carriers.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2b19f6b6-e1cd-4493-bb10-d08be3114c56.png)

The company meant to keep its material out of training collection. Its live-answer access was never part of the discussion. **Intent is not configuration:** the edge applies the rule it has, not the distinction the team had in mind.

### 2. The public `robots.txt` says something unexpected

Next, someone adds a broad `Disallow` rule, or a managed setting changes what an outside visitor receives from `/robots.txt`. The file in a repository or CMS can look permissive while the public edge response says otherwise. Community discussion of [Cloudflare’s managed AI crawler blocking](https://www.reddit.com/r/CloudFlare/comments/1cws4w5/heads_up_the_block_ai_scrapers_toggle_is/) is a reminder to inspect the delivered response rather than assume the source file is the whole story.

That distinction is easy to miss. A developer checks the local file, sees `User-agent: *` and `Allow: /`, and tells marketing that bots are welcome. The retrieval agent checks the live domain. It receives the live domain’s rules.

A broad refusal at this point can stop a visit before the pricing page, comparison table, or case study enters the conversation. **Check what the edge serves, not just what the team saved.**

### 3. The firewall challenges a permitted fetcher

Fixing `robots.txt` does not automatically fix the connection. A Web Application Firewall may still block a request or demand a browser challenge that a lightweight retrieval fetcher does not complete. The advisory file says yes; the edge response says no.

[DataDome’s analysis of Anthropic bots](https://datadome.co/bot-management-protection/claude-user-anthropic/) describes the difficulty of managing `Claude-User` through a fixed IP allowlist. [Perplexity’s crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) addresses access for its crawlers, including IP information administrators can use. These details matter when an operator tries to distinguish a verified fetcher from someone merely putting a familiar bot name in a request header. One [SignalWord audit](https://signalword.ai/) found 223 of 1,715 business sites where `robots.txt` permitted AI crawlers but the server or CDN refused access.

![Illustration of an invisible wall keeping an AI search cursor from website content.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/0c012dfd-53b3-411d-994c-efd83151cd9e.png)

This is the point where the incident becomes especially misleading. A browser visit works. The rules file appears open. Yet the fetch that matters can still end at the perimeter. **Permission in `robots.txt` is not proof of delivery.**

### 4. The page loads, but its answer does not

Some fetchers get through both gates and still receive little useful material. If pricing, product details, or customer proof appear only after client-side JavaScript runs, the initial HTML response may be an almost empty shell. A browser fills it in. A lightweight fetch may not.

That does not mean every JavaScript site is absent from every AI answer. It means the team cannot assume that what it sees after a browser renders a page is what a particular retrieval request received. For commercially important pages, inspect the **HTML delivered before client-side scripts run**. If the facts a buyer needs are missing there, another content campaign will not put them into that response.

## Damage: a full calendar, an empty path to the answer

By now, the marketing team has improved material the assistant may never have read. The dashboard keeps reporting missing mentions. Search Console remains reassuring. Engineering sees no outage because ordinary visitors can still load the site. Each team has a piece of a normal-looking picture.

Meanwhile, a competitor whose relevant page is reachable can supply the answer. I would not conclude that every missing citation is caused by bot blocking, or that access guarantees a citation. The narrower point is enough: **content cannot compete in a live fetch that never receives it.** Until the team tests that path, more publishing is a bet placed without looking at the table.

## Fix: separate training policy from answer access

The corrective decision is not “allow every crawler.” It is to decide what each kind of bot may do, then make the public rules and the firewall match that decision. [RunCloud’s discussion of bot management](https://runcloud.io/blog/block-ai-crawlers) describes the need to manage these layers rather than rely on a single switch.

For a business that chooses to disallow the training crawlers named here while permitting these search and user-triggered agents, a starting `robots.txt` distinction looks like this:

```text
# Disallow foundation-model training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Permit search and user-triggered retrieval
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /
```

This is **an advisory policy, not a firewall bypass**. It does not make a blocked HTTP request succeed, and it does not establish that every permitted bot will use a page in an answer. The operator still has to inspect CDN and WAF rules, verify the agents it intends to permit, and ensure those requests can receive useful HTML. Do not turn a bot-name string into an unconditional security exception; a request header alone is not verification.

I would run the checks in the order the fetch encounters them:

1. **Inspect the public edge response.** Request `/robots.txt` and a commercially important page from outside the application. Look at status codes, redirects, challenge indicators, and the text actually returned. Compare that text with the file the team believes it published.
2. **Review the CDN and WAF decision.** Find the rule behind a refusal or challenge, then check whether verified search and user-triggered fetchers can pass it. A permissive `robots.txt` cannot overrule a `403`.
3. **Read the served page.** Inspect its initial HTML for the pricing, capabilities, comparison details, or proof the assistant would need. If those facts depend on client-side rendering, address delivery before commissioning replacements for the prose.
4. **Use logs to confirm what changed.** Look for verified bot requests and their response codes. A request to `/robots.txt` establishes that the bot checked the door; a successful page response establishes more. Neither, by itself, promises a citation.

These quick curl requests help reveal what the public edge returns to requests bearing the named user-agent strings:

```bash
curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Claude-User/1.0" -I https://example.com/robots.txt
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" -I https://example.com/
```

Treat them as **a first check, not proof of verified-bot access**. The requests are yours, not Anthropic’s or OpenAI’s, and a WAF may treat them differently. Fetch the body of `/robots.txt` as well; `-I` shows headers, not the rules. Then pair the result with CDN decisions and verified-bot logs. An empty log search alone is not a diagnosis: the logging scope, retention window, and point of interception all matter.

![Technical checklist showing checks at the edge, robots.txt, and page-rendering layers.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a0539cc1-dd5f-4b19-a053-95f158995428.png)

The owner of each fix also matters. A CMS or web team can change the published `robots.txt` and put essential information into the initial HTML. A person with CDN or WAF access must change an edge rule or challenge. A mention tracker cannot make either change, however polished its chart looks.

At [groas](https://groas.com/), we treat organic AI visibility alongside paid search as an operational problem, not just a publishing target. That means examining crawler access and edge responses as well as the content a buyer might ask about. The [groas MCP integration for ChatGPT and Claude](https://groas.com/mcp) lets teams query live bot traffic and citation health from their workflow. It does not replace the person who controls a client’s firewall when a rule needs changing.

The rule I now apply is simple: **never block an AI crawler by category; filter it by function.** If you want training crawlers out, keep them out. Do not accidentally lock the door on the fetcher trying to read an answer for your buyer.

## Frequently Asked Questions

### Does good Google Search Console performance mean AI assistants can fetch my site?

No. A site can be fully available to human visitors and Google while a live-answer fetcher is challenged or refused at the edge. If the fetch fails, the assistant simply leaves the brand out of the answer, usually without notifying the marketing team.

### Are all AI crawlers the same, or do some matter more for being cited in answers?

They serve different functions. Training crawlers like GPTBot and ClaudeBot collect material for model development. Search crawlers like OAI-SearchBot and PerplexityBot help pages appear in search-based answers, and user-triggered fetchers like ChatGPT-User and Claude-User visit pages when a user asks for current information. Blocking by category can cut off the fetchers that produce live citations.

### My brand isn't mentioned in AI answers — should I publish more comparison content first?

Not before checking whether the assistant can retrieve the pages the company has already published. A tracker can show a brand was not mentioned, but it cannot tell whether a CDN returned a 403 Forbidden, an edge rule altered robots.txt, or the page arrived without its useful content. Content work does not help a fetch that never receives the page.

### My robots.txt allows bots in the codebase — could the public version still be blocking them?

Yes. A managed CDN setting or edge rule can change what an outside visitor receives from /robots.txt, so the delivered response may say something different from the file saved in the repository or CMS. A developer checking the local file can wrongly assure marketing that bots are welcome; the retrieval agent obeys the rules served by the live domain.

### Can a firewall block an AI fetcher even when robots.txt allows it?

Yes. A Web Application Firewall can refuse the request or demand a browser challenge that a lightweight retrieval fetcher does not complete, so the advisory file says yes while the edge response says no. One SignalWord audit found 223 of 1,715 business sites where robots.txt permitted AI crawlers but the server or CDN refused access.

### Can my pages load fine in a browser but still look empty to an AI fetcher?

Yes. If pricing, product details, or customer proof appear only after client-side JavaScript runs, the initial HTML response may be an almost empty shell. A browser fills it in; a lightweight fetch may not. For commercially important pages, inspect the HTML delivered before client-side scripts run.

### How should I fix AI bot blocking without accidentally cutting off AI answers?

Decide what each kind of bot may do, then make the published rules and firewall match that decision — for example, disallow training crawlers like GPTBot while permitting fetchers like Claude-User and OAI-SearchBot. Then inspect the public edge response for robots.txt and a key page, review CDN and WAF rules, read the served HTML for buyer-relevant facts, and confirm changes in verified bot logs. Treat any robots.txt change as advisory; it cannot overrule a 403.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#webpage","url":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q","name":"Postmortem: The Bot Block That Kept a Business Out of AI Answers","description":"A composite postmortem of how an AI-bot setting can block live answer fetchers, why content work misses the problem, and how to check the edge, robots.txt, and rendered page.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d773e603-2e35-4109-bf59-8eb3bd9fa4a5.png"},"breadcrumb":{"@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#breadcrumb"},"dateModified":"2026-10-16T05:58:05.504Z","mainEntity":{"@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Postmortem: The Bot Block That Kept a Business Out of AI Answers","item":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#article","url":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q","mainEntityOfPage":{"@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Postmortem: The Bot Block That Kept a Business Out of AI Answers","description":"A composite postmortem of how an AI-bot setting can block live answer fetchers, why content work misses the problem, and how to check the edge, robots.txt, and rendered page.","datePublished":"2026-10-16T05:58:05.414Z","dateModified":"2026-10-16T05:58:05.504Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d773e603-2e35-4109-bf59-8eb3bd9fa4a5.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"robots txt, bot, robots, txt, answer, page, block, content","wordCount":2025,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/postmortem-how-one-bot-blocking-toggle-q#faq","mainEntity":[{"@type":"Question","name":"Does good Google Search Console performance mean AI assistants can fetch my site?","acceptedAnswer":{"@type":"Answer","text":"No. A site can be fully available to human visitors and Google while a live-answer fetcher is challenged or refused at the edge. If the fetch fails, the assistant simply leaves the brand out of the answer, usually without notifying the marketing team."}},{"@type":"Question","name":"Are all AI crawlers the same, or do some matter more for being cited in answers?","acceptedAnswer":{"@type":"Answer","text":"They serve different functions. Training crawlers like GPTBot and ClaudeBot collect material for model development. Search crawlers like OAI-SearchBot and PerplexityBot help pages appear in search-based answers, and user-triggered fetchers like ChatGPT-User and Claude-User visit pages when a user asks for current information. Blocking by category can cut off the fetchers that produce live citations."}},{"@type":"Question","name":"My brand isn't mentioned in AI answers — should I publish more comparison content first?","acceptedAnswer":{"@type":"Answer","text":"Not before checking whether the assistant can retrieve the pages the company has already published. A tracker can show a brand was not mentioned, but it cannot tell whether a CDN returned a 403 Forbidden, an edge rule altered robots.txt, or the page arrived without its useful content. Content work does not help a fetch that never receives the page."}},{"@type":"Question","name":"My robots.txt allows bots in the codebase — could the public version still be blocking them?","acceptedAnswer":{"@type":"Answer","text":"Yes. A managed CDN setting or edge rule can change what an outside visitor receives from /robots.txt, so the delivered response may say something different from the file saved in the repository or CMS. A developer checking the local file can wrongly assure marketing that bots are welcome; the retrieval agent obeys the rules served by the live domain."}},{"@type":"Question","name":"Can a firewall block an AI fetcher even when robots.txt allows it?","acceptedAnswer":{"@type":"Answer","text":"Yes. A Web Application Firewall can refuse the request or demand a browser challenge that a lightweight retrieval fetcher does not complete, so the advisory file says yes while the edge response says no. One SignalWord audit found 223 of 1,715 business sites where robots.txt permitted AI crawlers but the server or CDN refused access."}},{"@type":"Question","name":"Can my pages load fine in a browser but still look empty to an AI fetcher?","acceptedAnswer":{"@type":"Answer","text":"Yes. If pricing, product details, or customer proof appear only after client-side JavaScript runs, the initial HTML response may be an almost empty shell. A browser fills it in; a lightweight fetch may not. For commercially important pages, inspect the HTML delivered before client-side scripts run."}},{"@type":"Question","name":"How should I fix AI bot blocking without accidentally cutting off AI answers?","acceptedAnswer":{"@type":"Answer","text":"Decide what each kind of bot may do, then make the published rules and firewall match that decision — for example, disallow training crawlers like GPTBot while permitting fetchers like Claude-User and OAI-SearchBot. Then inspect the public edge response for robots.txt and a key page, review CDN and WAF rules, read the served HTML for buyer-relevant facts, and confirm changes in verified bot logs. Treat any robots.txt change as advisory; it cannot overrule a 403."}}]}]}
```