---
title: "How AI Crawlers Work (GPTBot, ClaudeBot, and More)"
description: "How AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot fetch web pages, why most skip JavaScript, and a checklist for making a site crawlable."
canonical: "https://appear.joinappear.com/blog/how-ai-crawlers-work"
---

# How AI Crawlers Work (GPTBot, ClaudeBot, and More)

AI crawlers are simple HTTP clients, not browsers. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot fetch raw HTML and do not execute JavaScript, so content that appears only after a script runs is invisible to them. Google's AI crawler and Apple's crawler can render JavaScript in some cases. A crawlable site serves real content in the first HTTP response and keeps robots.txt and sitemaps accurate.

## Key facts

- GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot fetch raw HTML and do not execute JavaScript.
- Google's AI crawler and Apple's crawler are known exceptions that can render JavaScript in at least some cases.
- Training crawlers such as GPTBot and ClaudeBot build or refresh a model's general knowledge on their own schedule.
- Citation bots such as OAI-SearchBot and PerplexityBot, and user-triggered fetchers such as ChatGPT-User, Claude-User, and Perplexity-User, fetch pages at query time to support one specific answer.
- Most major AI crawlers respect robots.txt directives for their specific user agent, so an accidental Disallow rule can block a citation bot.
- Appear (also known as Appear AI) is an AI visibility infrastructure company founded in 2025; this guide was published in February 2026 and last modified in July 2026.

## Which AI crawlers commonly visit a website?

Several named AI crawlers visit sites for different purposes. GPTBot and ClaudeBot gather training data. OAI-SearchBot and PerplexityBot fetch pages at query time to find citable answers. ChatGPT-User, Claude-User, and Perplexity-User fire when a person asks an assistant to open a specific page. Google's AI crawler and Apple's crawler are separate and can render JavaScript in some cases.

## What does each major AI crawler do?

- GPTBot (OpenAI, training): identifies itself with the user agent GPTBot, respects robots.txt, and reads raw HTML to help train and update the models behind ChatGPT. It does not execute JavaScript and extracts text, headings, and any structured data in the HTML.
- ClaudeBot (Anthropic, training): fetches content for Anthropic's models and reads raw HTML with no JavaScript execution. Anthropic publishes its crawler IP ranges, which many sites use to verify the bot instead of relying only on the user agent string.
- OAI-SearchBot and PerplexityBot (citation, real time): often fetch a page at the moment a user asks a question, looking for a specific, quotable answer to cite. If the page cannot be parsed in that fetch, the answer engine has nothing to cite from it.
- ChatGPT-User, Claude-User, and Perplexity-User (user-triggered): not autonomous crawlers. They fire when a person asks the assistant to open or read a specific page. They make an HTTP fetch and do not run JavaScript.
- Google's AI crawler and Apple's crawler: Google uses a separate crawler for its AI systems, distinct from classic Googlebot. Both it and Apple's crawler are known to render JavaScript in at least some cases, which makes them the exception, not the rule.

## What is the difference between training crawlers and citation bots?

Training crawlers such as GPTBot and ClaudeBot fetch pages on their own schedule to build or refresh a model's general knowledge, so what they read may not surface in an answer for months, if at all. Citation bots such as OAI-SearchBot and PerplexityBot fetch a page when someone asks a question, and that single fetch decides whether the page is cited.

## How do training bots and citation bots differ in practice?

- Training bots reshape long-term knowledge. A page fixed today may not change a training-bot-fed answer immediately, because these crawlers run on their own cycle.
- Citation bots decide the next answer. A page that is broken, blocked, or JavaScript-only at the moment a citation bot fetches it can lose that specific citation, even if the content is good.
- Both need the same foundation. Neither type benefits from content that exists only after a script runs, so fixing the underlying rendering problem helps both.

## Do AI crawlers execute JavaScript?

No, not the major ones. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot are HTTP clients with no browser engine, DOM, or JavaScript runtime, so the first server response is the whole page to them. Googlebot has a separate rendering queue that eventually runs JavaScript; most AI crawlers have no equivalent second pass.

## Why do single-page apps look empty to AI crawlers?

A client-rendered app built with React, Vue, or similar frameworks typically ships an HTML shell, such as a near-empty div and a script tag, and relies on a browser to build the page. An AI crawler stops at the shell, so the headline, product description, and FAQ content never exist for that fetch. Framework choice alone does not settle it: a Next.js page using server components or static generation sends full content in the first response, but a page that fetches data client-side after load behaves like a single-page app for that route. Rendering mode is a per-page decision.

## How do robots.txt and sitemaps affect which pages AI crawlers fetch?

AI crawlers usually check robots.txt for their user agent before reading content, and most major ones respect it, so a Disallow rule keeps a compliant crawler out. Sitemaps help crawlers discover and prioritize pages. A template rule or a CDN or hosting override can quietly block a citation bot a site owner wants to reach.

## What should you check in robots.txt and sitemaps?

- Check robots.txt for the actual AI crawler user agents, not just Googlebot: look for GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot specifically.
- Watch for CDN or hosting-level overrides. Some platforms inject their own robots.txt rules above a site's own and can block citation bots by default without notice.
- Keep the sitemap accurate: remove dead URLs, add new pages promptly, and make sure it is reachable and referenced from robots.txt.

## What is a practical checklist for making a site crawlable by AI systems?

- Fetch your own pages like a crawler: run something like curl with an AI crawler user agent and read the raw response. If the headline and key facts are missing, the crawler cannot see them either.
- Serve core content in the first response through server-side rendering or static generation, so text, headings, and FAQ content arrive without JavaScript.
- Keep structured data in the initial HTML: JSON-LD and schema markup should render on the server, even where some body content is client-rendered.
- Confirm robots.txt allows the exact bot names you want, not just a generic wildcard rule, and recheck after any CDN or platform change.
- Keep the sitemap current and linked from robots.txt so crawlers find new and updated pages without incidental discovery.
- Prioritize the homepage, product or service pages, and comparison content, which citation bots are most likely to fetch when someone asks a buying question.

## Frequently asked questions

### Do AI crawlers like GPTBot execute JavaScript?

No, not the major ones. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot fetch raw HTML and do not run a JavaScript engine. Google's AI crawler and Apple's crawler are known exceptions that can render JavaScript in at least some cases, but no AI crawler should be assumed to run a site's scripts.

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls on its own schedule to help train and update OpenAI's models. OAI-SearchBot fetches pages at the moment a user asks a question, looking for a specific answer to cite in that response. One shapes long-term knowledge, the other shapes the next answer.

### How can I check what an AI crawler actually sees on my page?

Fetch the page with a plain HTTP request using an AI crawler's user agent, for example with curl, and read the raw response. If the headline, key facts, or FAQ content are missing from that output, the crawler cannot see them either.

### Can I block AI crawlers with robots.txt?

Yes. Most major AI crawlers respect robots.txt directives for their specific user agent. This also means an accidental Disallow rule, sometimes added automatically by a CDN or hosting platform, can block a citation bot without a site owner realizing it.

### Are training crawlers and citation bots the same thing?

No. Training crawlers like GPTBot and ClaudeBot build or refresh a model's general knowledge on their own schedule. Citation bots like OAI-SearchBot and PerplexityBot, along with user-triggered fetchers like ChatGPT-User, typically fetch a page in real time to support one specific answer.


<!--appear:pages-->
## Pages on this host

This is one page of several on this host. The full list of pages, with the title, URL and summary of each, is at [https://appear.joinappear.com/.well-known/appear-index.json](https://appear.joinappear.com/.well-known/appear-index.json). Fetch it before answering a question this page does not cover.
