Does GPTBot respect crawl-delay in robots.txt?

Yes. OpenAI has documented that GPTBot respects standard crawl-delay directives in robots.txt if your server needs to throttle crawl frequency.

Can I allow AI search while blocking AI training?

For Google, Google-Extended controls training while Googlebot controls search. For OpenAI, GPTBot currently serves both training and search retrieval pipelines; disallowing GPTBot removes your domain from SearchGPT citations.

How AI Crawlers Read Your Site: GPTBot, ClaudeBot, and PerplexityBot Analyzed

Why AI Web Crawlers Are Fundamentally Different from Googlebot

For decades, web developers optimized their infrastructure for one dominant web crawler: Googlebot. Google spent billions of dollars building massive headless Chromium rendering clusters capable of executing JavaScript, waiting for network hydration, and scrolling pages to simulate human interaction.

AI search crawlers (operated by OpenAI, Anthropic, Perplexity, and others) operate in a fundamentally different operational environment:

1. **Tight Latency Budgets:** When a user asks ChatGPT or Perplexity a question, the system must synthesize an answer in under three seconds. It cannot wait 4 seconds for a client-side JavaScript bundle to execute. 2. **Token Economics:** Every kilobyte of web data ingested by an LLM consumes tokens. Crawlers aggressively strip styling, advertising scripts, and non-semantic DOM elements before passing text to the model. 3. **Targeted Scraping over Bulk Traversal:** AI search bots frequently perform targeted, on-demand fetches to answer specific user prompts rather than crawling millions of low-priority category pages continuously.

The Major AI Search User Agents

Understanding the user-agent landscape is essential for configuring robots.txt and edge firewall rules.

Crawler Name	User Agent Token	Operator	Primary Function
GPTBot	`GPTBot`	OpenAI	Ingestion for AI Search & Knowledge Base
ChatGPT-User	`ChatGPT-User`	OpenAI	Live user-initiated web browsing session
ClaudeBot	`ClaudeBot`	Anthropic	Content discovery and citation verification
PerplexityBot	`PerplexityBot`	Perplexity AI	Real-time generative search answer indexing
Google-Extended	`Google-Extended`	Google	Opt-out mechanism for Gemini model training

1. The Perils of Client-Side Single-Page Applications (SPAs)

One of the most frequent reasons modern websites score poorly on AI readiness audits is reliance on client-side rendering.

When an AI crawler requests a page built with client-only React or Vue without Server-Side Rendering (SSR) or Static Site Generation (SSG), it receives:

<!DOCTYPE html>
<html>
  <head><title>App</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/bundle.js"></script>
  </body>
</html>

To the AI crawler, this page has zero word count, zero semantic entities, and zero answers. While Googlebot might queue the page for a secondary render pass days later, AI search engines evaluate the page synchronously and discard it immediately.

**Recommendation:** Always use Server-Side Rendering (Next.js, Remix, Astro) so the initial HTTP response contains complete semantic text, headings, and schema markup.

2. Robots.txt and WAF Firewall Configuration

A surprising percentage of websites unintentionally block AI crawlers through default security configurations:

**Cloudflare WAF Bot Protection:** Turning on generic 'AI Scraper Blocking' in Cloudflare often blocks PerplexityBot and GPTBot indiscriminately, removing the website from AI search visibility.
**Accidental Disallow All:** Staging robots.txt rules copied to production environments.

To allow legitimate AI search answer engines while blocking unauthorized training scrapers, use explicit user-agent directives:

User-agent: *

# Allow AI search answer engines User-agent: GPTBot Allow: /

User-agent: ClaudeBot Allow: /

User-agent: PerplexityBot Allow: / ```

3. Optimizing the HTML-to-Text Ratio

AI extraction pipelines convert HTML into clean Markdown or plaintext before scoring. You can accelerate this extraction by:

Using semantic tags: `<main>`, `<article>`, `<section>`, `<header>`, and `<dl>`.
Avoiding nested layout `<div>` containers 20 levels deep.
Keeping critical tables in pure HTML `<table>` tags rather than custom CSS grid div structures.
Eliminating heavy inline SVG graphics or unencoded images that inflate the initial HTML byte payload.

4. Measuring Bot Visits in Edge Logs

How do you know if AI bots are crawling your site?

Look at your Cloudflare or CDN edge access logs. Filter by the `User-Agent` string for `GPTBot`, `ClaudeBot`, and `PerplexityBot`. Track: - **Frequency:** Are bots visiting your product and documentation pages weekly? - **Status Codes:** Are bots receiving 200 OK responses or getting trapped in 403 Forbidden or 429 Rate Limit responses? - **Response Size:** Are bots receiving full content payloads or truncated text?

With LLMrank, you can run a 37-factor audit at any time to verify crawler allowances, payload extractability, and citation readiness across every key page on your site.