# Can AI Crawlers Read Your Site? GPTBot & Others

> The main AI crawlers, how robots.txt affects them, why JavaScript-heavy sites get omitted from AI summaries, and how to verify access yourself.

URL: https://aiseoperth.net.au/guide/can-ai-crawlers-read-your-site/
Last-Modified: 2026-07-29

process guide

# Can AI Crawlers Actually Read Your Site?

The main AI crawlers, how robots.txt affects them, why JavaScript-heavy sites get omitted from AI summaries, and how to verify access yourself.

Published 29 July 2026 7 min read

![Multiple access paths to a structure, some open and some closed](/images/featured/multiple-access-paths-leading-to-one-structure-wit.webp)

We see one recurring issue dominating technical audits in technical audits. Managing ai crawler access robots txt files is frequently mishandled, and the visibility problem remains completely invisible until someone actively looks for it.

This happens because organisations misunderstand how these new agents operate.

The check itself takes about twenty minutes to run. 

An AI SEO audit

[/llm-visibility-audit/ →](/llm-visibility-audit/)

 completes this check per individual agent rather than inferring your status from a single line of code.

So, we will break down the specific crawlers you need to know about right now. This guide covers the exact access settings and rendering fixes required to keep your content visible, and it is the first layer of 

AI SEO in Perth

[/ →](/)

 because nothing above it works while access is blocked.

## The crawlers that matter, and what each does

The most critical AI crawlers for business visibility in 2026 are GPTBot, ClaudeBot, and Google-Extended. These three agents dictate exactly how your brand appears across major platforms like ChatGPT and Google’s AI Overviews.

The table below maps the primary operators and what each one is actually for.

![Table-like grid of crawler entries with open and closed access states](/images/content/clean-table-like-grid-of-crawler-entries-with-open.webp)

| User agent | Operator | Primary purpose |
| --- | --- | --- |
| GPTBot | OpenAI | Crawling content for model training |
| OAI-SearchBot | OpenAI | Indexing for search-style answers |
| ChatGPT-User | OpenAI | Fetching a page in response to a user request |
| ClaudeBot | Anthropic | Crawling content for model training |
| Claude-SearchBot | Anthropic | Indexing for search results within Claude |
| Claude-User | Anthropic | Fetching a page in response to a user request |
| PerplexityBot | Perplexity | Indexing for search results |
| Perplexity-User | Perplexity | Fetching a page in response to a user request |
| Google-Extended | Google | Controls use of content for Gemini and AI Overviews generation |
| Bytespider | ByteDance | Crawling for training and product use |
| CCBot | Common Crawl | Building the open crawl corpus many models train on |

The names change frequently and new ones appear constantly. Check the official operator documentation directly rather than trusting any static list.

Automated traffic is now a large share of what hits any public site, and AI agents are a growing part of that share.

> These crawlers read far more pages than they send visitors back. That asymmetry is real, and it is a fair reason to make access a deliberate decision rather than a default one.

Some agents are more aggressive than others, which is a fair reason to reconsider blanket access. ByteDance operates Bytespider, which scrapes heavily for training and has a poor reputation for crawler etiquette.

## Training access and retrieval access are separate decisions

Training access allows an AI to use your content to build its foundational model, while retrieval access allows it to cite your live page to answer a user query today. When clients ask us, “should i block ai crawlers?”, the correct answer always starts by separating these two functions.

The most expensive mistake a business can make is accidentally blocking retrieval agents when they only intended to block training. Our technical audits reveal this error constantly. Different user agents govern each function, and blocking one bot absolutely does not block the other.

During 2023, a massive wave of businesses added blanket disallow commands in response to the early model training debates. Many of those companies wanted the training restriction but failed to realise they were also removing their websites from live retrieval.

This accidental blocking creates three distinct problems:

-   Your brand vanishes from real-time AI tool recommendations.
-   Competitors gain direct access to your previous search market share.
-   Your site loses highly qualified, high-intent referral traffic.

Retrieval is the surface that produces shortlists and recommendations for local customers. Other businesses blocked a single named agent and assumed the whole AI category was covered. Both mistakes are common, and both are invisible until someone checks.

> Training access and retrieval access are separate decisions. Blocking broadly is a choice you can defend. Making it by accident is not.

Either restriction is defensible depending on your revenue model. A publisher holding a licensing position has a strong argument for restricting training access. A local service business trying to appear in an assistant’s recommendation list has almost no reason to restrict retrieval. The point is to make one decision without getting the other outcome by accident.

## Reading and writing robots.txt for these agents

Managing your ai crawler access robots txt configuration requires extreme precision. Directives in the file are matched per specific user-agent block, and the most specific matching block always wins. A wildcard block does not apply to any agent that has its own named block. Most misconfigurations trace back to that single rule.

```
# Allow retrieval, restrict training
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Everything else
User-agent: *
Allow: /
```

The major mistake to avoid is placing a `Disallow: /` command under the generic `User-agent: *` section while assuming named agents will follow a more permissive rule placed elsewhere. You must configure a specific gptbot robots txt directive if you want precise control over OpenAI’s agents. Bots strictly follow their own named block if one exists, and they default to the wildcard rule if one does not.

> Robots files have grown long. Many established sites now name a dozen or more AI agents explicitly, which means a copied template almost certainly does not say what you think it says.

This complexity proves that copying a generic file layout is no longer sufficient. You must also check your page code for a stray `noindex` meta tag or an `X-Robots-Tag` HTTP header. Those commands operate at a completely different technical layer.

They can actively remove pages from search results even when your text file explicitly permits access. Our auditing process always verifies these elements together. Aggressive scrapers might ignore polite text file requests entirely, meaning you must rely on server-side rules for true protection.

## The rendering problem, which is bigger than robots.txt

Googlebot renders JavaScript, but the vast majority of AI retrieval crawlers do not. If your website relies heavily on client-side frameworks, these agents will fail to process your information.

![Two contrasting page representations, one dense with content and one almost empty](/images/content/two-contrasting-page-representations-side-by-side-.webp)

This means a site built as a single-page application can deliver a nearly empty document to an agent that fetches it. The page looks perfectly fine to a human scrolling in a browser.

The AI crawler receives an empty shell. It is an invisible barrier, and it is common on ecommerce and app-style sites where the framework builds everything client-side.

| AI Crawler | JavaScript Execution | Citation Impact on Client-Side Sites |
| --- | --- | --- |
| Googlebot | Yes (Headless Chrome) | Full content indexed |
| Google-Extended | Yes (Inherits Google WRS) | Full content indexed |
| GPTBot | No | Receives blank page |
| ClaudeBot | No | Receives blank page |
| PerplexityBot | No | Receives blank page |

This technical gap explains why sites that rank respectably in standard Google Search can be completely absent from assistant answers. The traditional Google path works because Google actively renders the code. The AI retrieval path fails because the bots skip the rendering step. Assume the agent fetching your page executes no JavaScript at all, then check whether your content survives that assumption.

To test your own site’s rendering, follow these rapid steps:

-   Open your most valuable service page in a standard desktop browser.
-   Right-click and select “View Page Source” to bypass the rendered DOM.
-   Search the raw code for a distinctive sentence from your main content.

If the text is missing from that raw view, a non-rendering crawler is definitely not seeing it either. We advise clients to implement server-side rendering or static generation for all core content pages.

That shift represents a serious engineering decision with real financial cost, which is why finding out early matters. It is also frequently the root cause behind 

pages not appearing in AI Overviews

[/guide/why-your-pages-are-not-appearing-in-ai-overviews/ →](/guide/why-your-pages-are-not-appearing-in-ai-overviews/)

. Retrieval crawlers are fast precisely because they skip rendering. Speed is the trade they made, and your content is what pays for it.

## A verification sequence you can run today

You can verify your AI crawler access by auditing your text file, viewing your raw page source, and checking your server headers. Testing these elements confirms exactly what external agents see when they ping your domain. Our security engineers use this exact sequence to isolate visibility failures.

1.  **One.** Open `yoursite.com/robots.txt` and carefully read every single block. Note which named agents appear and what each is explicitly allowed to access.
2.  **Two.** View the raw source code on your homepage and on one high-value service page. Search for a distinctive phrase from the visible content. Present means non-rendering crawlers can read it; absent means they cannot. Using a tool like `cURL` provides the most accurate view.
3.  **Three.** Check your server response headers for an `X-Robots-Tag`, and inspect your page source for a `noindex` meta tag.
4.  **Four.** Check your CDN or WAF rules. Products like Cloudflare often block unfamiliar user agents by default, which produces exactly the same negative outcome as a text file disallow but with zero visibility.
5.  **Five.** Check your raw server logs for the specific agent names listed above. Presence confirms they are reaching your infrastructure; absence over several weeks strongly suggests something upstream is stopping them.

If steps two and five disagree with what your text file says should be happening, the root cause is almost always step four. We frequently find aggressive firewall settings overriding perfectly good crawler configurations. Fixing that one setting restores your entire AI search presence instantly.

## Deciding deliberately

You must document a deliberate policy specifying which agents are allowed, which are restricted, and the exact business reasoning behind those choices. Setting up your ai crawler access robots txt rules properly is no longer just a technical task; it is a critical business strategy. Write the decision down and review it quarterly.

For most standard companies, the sensible default is allowing retrieval agents while making a completely separate calculation regarding training bots. Publishers, membership sites, and businesses whose written content is the actual product face a very different equation.

> Restricting all access is a highly legitimate choice for those specific content-driven business models. What is entirely indefensible is simply not knowing your current status.

We outline the deep technical details of what follows once access is confirmed in 

what an audit covers

[/guide/what-an-ai-seo-audit-covers/ →](/guide/what-an-ai-seo-audit-covers/)

. The specific tactics for improving your performance on conversational surfaces are fully documented in our guide to 

ChatGPT optimisation

[/chatgpt-optimisation/ →](/chatgpt-optimisation/)

. Take control of your crawler directives today to protect your assets and secure your future search traffic.

Questions

## Common questions

### Should I block GPTBot?

It depends on whether you want your content used for training versus retrievable for answers, and those are governed by different agents. Blocking broadly can remove you from a surface your buyers use, so it is worth deciding deliberately rather than by default.

### Does blocking Google-Extended affect normal Google rankings?

It governs use of your content for Gemini and AI Overviews generation rather than standard Search indexing. That distinction has shifted before, so verify it against Google's current documentation before acting.

### How do I check what a crawler sees?

Fetch the page without JavaScript execution and compare it to the rendered version. If the main content only exists after rendering, many retrieval crawlers will not see it. Viewing source in a browser shows you the raw document.

### Do these crawlers respect robots.txt?

The major ones from established providers publish their user-agent strings and state that they respect robots.txt. Some smaller and less reputable crawlers do not, which is a separate problem from the visibility one and needs server-level handling.

## Want this handled properly?

AI SEO Audit is the engagement that puts this guide into practice. A free consultation covers whether it fits your business.

Learn more about AI SEO Audit

[/llm-visibility-audit/ →](/llm-visibility-audit/)

 

Book a free consultation

[/contact/ →](/contact/)