Controlling GPTBot, Perplexity & Claude Crawlers via robots.txt

Executive Overview: Controlling AI Search Bots & Scraping Crawlers

In the modern web landscape, artificial intelligence companies deploy automated crawlers to harvest massive datasets for Large Language Model (LLM) training and real-time Retrieval-Augmented Generation (RAG). While traditional search crawlers like Googlebot and Bingbot index web pages to send referral traffic back to publishers, AI scrapers often digest content to display direct answers—frequently bypassing publisher visits altogether.

To maintain control over intellectual property, bandwidth costs, and server resources without sacrificing search engine visibility, technical SEOs and site administrators must implement a multi-layered defense strategy. This guide explores precise robots.txt directives, HTTP response header controls (X-Robots-Tag), CDN-level WAF rules, and Next.js Edge Middleware patterns.

---

Understanding AI Crawler Taxonomies & User-Agents

Not all AI bots perform the same function. AI crawlers fall into two distinct operational categories:

1. Offline Training Crawlers: Background bots that crawl billions of pages to train future foundational models (e.g., GPT-5, Claude 3.5 Sonnet).
2. Real-Time Retrieval Crawlers: Bots invoked on-demand when a user asks an AI search engine (e.g., ChatGPT Search, Perplexity) to summarize a live web page.

Comprehensive AI Crawler User-Agent Reference Matrix

---

Production-Grade robots.txt Configuration

A well-structured robots.txt file acts as the first line of defense. By targeting specific User-Agent strings, you can grant access to search engines while restricting AI scrapers.

User-Agent: Googlebot
Allow: /

User-Agent: Bingbot
Allow: /

User-Agent: DuckDuckBot
Allow: /

User-Agent: YandexBot
Allow: /

User-Agent: GPTBot
Disallow: /

User-Agent: ChatGPT-User
Allow: /

User-Agent: ClaudeBot
Disallow: /

User-Agent: Claude-Web
Allow: /blog/
Allow: /docs/
Disallow: /api/
Disallow: /private/
Crawl-delay: 2

User-Agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-Agent: CCBot
Disallow: /

User-Agent: Bytespider
Disallow: /

User-Agent: Meta-ExternalAgent
Disallow: /

User-Agent: Diffbot
Disallow: /

Sitemap: https://www.seotech.app/sitemap.xml

---

Advanced Edge Middleware & HTTP Header Enforcement

Because robots.txt is an advisory standard that non-compliant crawlers may bypass, modern web applications enforce bot management at the application layer or CDN edge.

HTTP Response Headers (X-Robots-Tag)

The X-Robots-Tag header provides explicit metadata instructions to crawlers processing non-HTML files (like PDF downloads, JSON endpoints, or images):

HTTP/1.1 200 OK
Content-Type: application/json
X-Robots-Tag: noindex, noarchive, noai, noimageai

Next.js Edge Middleware Bot Guard

Below is a production Next.js Edge Middleware script that inspects incoming user agents, enforces HTTP 403 blocks for unauthorized scrapers, and injects protective security headers.

import { type NextRequest, NextResponse } from 'next/server'

// Targeted AI Scraper User-Agent Signatures
const BLOCKED_AI_SCRAPERS = [
  'GPTBot',
  'ClaudeBot',
  'CCBot',
  'Bytespider',
  'Meta-ExternalAgent',
  'Diffbot',
  'Cohere-ai',
  'Omgilibot',
]

// Real-Time Search Crawlers Allowed for Live Citations
const ALLOWED_RETRIEVAL_BOTS = ['ChatGPT-User', 'PerplexityBot', 'Claude-Web']

export function middleware(request: NextRequest) {
  const userAgent = request.headers.get('user-agent') || ''
  const pathname = request.nextUrl.pathname

  // Allow static assets and Next.js internal files immediately
  if (
    pathname.startsWith('/_next') ||
    pathname.startsWith('/favicon.ico') ||
    pathname.match(/\.(png|jpg|jpeg|svg|webp|css|js)$/)
  ) {
    return NextResponse.next()
  }

  // Check if request comes from an allowed real-time retrieval bot
  const isAllowedRetrieval = ALLOWED_RETRIEVAL_BOTS.some((bot) =>
    userAgent.toLowerCase().includes(bot.toLowerCase())
  )

  if (isAllowedRetrieval) {
    const response = NextResponse.next()
    response.headers.set('X-AI-Retrieval-Status', 'Allowed-Live-Search')
    return response
  }

  // Check for unauthorized AI training bots
  const isBlockedScraper = BLOCKED_AI_SCRAPERS.some((bot) =>
    userAgent.toLowerCase().includes(bot.toLowerCase())
  )

  if (isBlockedScraper) {
    return new NextResponse(
      JSON.stringify({
        error: 'Forbidden',
        message: 'AI model training scrapers are restricted on this domain.',
        documentation: 'https://www.seotech.app/blog/ai-search-bot-indexing-control',
      }),
      {
        status: 403,
        headers: {
          'Content-Type': 'application/json',
          'X-Robots-Tag': 'noindex, noarchive, noai',
        },
      }
    )
  }

  // Standard response with default AI protection headers
  const response = NextResponse.next()
  response.headers.set('X-Robots-Tag', 'noai, noimageai')
  return response
}

export const config = {
  matcher: '/((?!api/health).*)',
}

---

Cloudflare WAF & Edge Firewall Configuration

For enterprise applications experiencing heavy crawler traffic, filtering requests at the application layer (Node.js/Next.js) can incur unnecessary compute costs. Configuring Cloudflare Web Application Firewall (WAF) rules offloads bot blocking directly to the CDN edge.

Recommended Cloudflare Expression Rule

(http.user_agent contains "GPTBot" and not http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot")

Action: Block or Managed Challenge

---

Key Execution & Audit Checklist

• Audit Case-Sensitive User Agents: Confirm that robots.txt strings match exact official bot definitions (GPTBot, ClaudeBot).
• Separate Training Bots from Retrieval Bots: Ensure ChatGPT-User and PerplexityBot are allowed if real-time AI citation traffic is desired.
• Deploy X-Robots-Tag Headers: Inject X-Robots-Tag: noai, noimageai across static file distributions and API routes.
• Verify Googlebot Indexability: Run Google Search Console URL Inspection to verify Googlebot is unaffected by AI bot rules.
• Monitor Server Compute Metrics: Track CPU and memory spikes to confirm that edge WAF rules successfully deflect brute-force scrapers.

---

Conclusion

Controlling AI crawlers requires a nuanced approach: blocking aggressive training bots while keeping channels open for referral-generating AI search engines. By combining strict robots.txt rules with CDN-level WAF filtering and X-Robots-Tag response headers, engineering teams can safeguard content assets without sacrificing organic search authority.

---

Official References

• →OpenAI GPTBot Specification & IP Ranges
• →Anthropic ClaudeBot & Claude-Web Documentation
• →Perplexity AI PerplexityBot Specification
• →Google Search Central - Robots.txt Rules & User-Agent Parsing
• →Cloudflare WAF Custom Rules & Bot Management
• →IETF RFC 9309 - Robots Exclusion Protocol Standard