GPTBot vs. ClaudeBot vs. PerplexityBot: How Each Actually Crawls JS Sites

In every test that's been run so far, none of the three execute JavaScript - that's a dated, repeated observation, not a permanent guarantee any provider has made. What's usually missing from "GPTBot vs. ClaudeBot vs. Perplexity" content is what actually differs between them, and how fast that's changing. This page uses three separate, clearly dated data sets rather than blending them into one falsely-current table - a static December 2024 snapshot and a May 2024-to-May 2025 Cloudflare comparison aren't the same kind of number, and treating them like they are is exactly the kind of imprecision this page exists to avoid.

The Comparison

Absolute volume snapshot - Vercel, December 2024

Useful for scale, dated honestly. This is the most-cited figure in this space, and it's over a year old.

These volume totals and 404 rates are network-wide figures - across Vercel's own network during the measured month - not a claim about any specific site's traffic mix. Individual sites, industries, and hosting stacks see very different bot mixes; see the prioritization section below for why that matters more than this table does.

Growth and share - Cloudflare, July 2025 (May 2024 to May 2025)

Cloudflare's report actually runs two different cohorts, and they give different share percentages for the same bots - worth showing both rather than picking whichever number sounds cleaner. "AI-only" is GPTBot/ClaudeBot/PerplexityBot/etc. measured against each other; "broader cohort" adds 30+ crawlers including Googlebot and Bingbot into the same denominator, which dilutes every AI bot's share. The comparison itself runs May 2024 to May 2025 - April 2025 shows up in Cloudflare's data only as Googlebot's single-month peak (crawling traffic 145% higher than May 2024), not as the comparison's endpoint.

The rest of this page uses the broader-cohort numbers by default, since that reflects a bot's real share of everything hitting a site, AI and non-AI combined - flagged explicitly wherever it matters. These are Cloudflare's own aggregate cohort shares, useful as background trend context - not a stand-in for any individual site's actual traffic mix, which is the point the prioritization section below makes explicitly.

Traffic by purpose - Cloudflare, August 2025

A separate, more specific Cloudflare analysis classifies AI-bot crawler traffic into four purpose categories: Training, Search, User action, and Undeclared. In the period Cloudflare illustrates, Training accounted for nearly 80% of all AI-bot crawling, while User action and Undeclared together made up less than 5% - leaving Search as most of the remainder. That's Cloudflare's own observed traffic-purpose split for the period shown, not a permanent or universal ratio, and "the rest is Search" is our own calculation from Cloudflare's two stated figures, not a number Cloudflare states directly. Search-index crawling (building or refreshing an index ahead of time) and live User-action traffic (a bot fetching a page because a person asked a question right now) are operationally different - see the taxonomy note below before treating either as "the retrieval bucket."

GPTBot (OpenAI)

In Vercel's December 2024 snapshot, GPTBot generated the most requests of any AI crawler measured across Vercel's network that month - 569 million - and it's a training crawler: OpenAI's own documentation says it "crawl[s] content that may be used in training" OpenAI's models, not that it does. Crawler access doesn't establish whether any specific page ends up in a training dataset or is later reflected in a model's output. On nextjs.org specifically, GPTBot prioritized HTML over other content types (57.70% of its fetches there), downloaded JavaScript files in roughly 11.5% of requests, and - in Vercel's tests - never executed them. Since then it's only gotten bigger by Cloudflare's measure: GPTBot's request volume rose 305% from May 2024 to May 2025 - its share of Cloudflare's broader crawler cohort (AI and search bots combined) grew from 2.2% to 7.7%, the fastest-growing major crawler of the group in that window. (Measured only against other AI crawlers, its share is much higher - 5% to 30% - the gap is entirely about which denominator you use, not two different facts.)

GPTBot is not the same agent as its search-facing sibling. OAI-SearchBot is OpenAI's own separate, automatic search crawler - it isn't documented as fetching a page live for each individual query; it crawls content ahead of time so a site can be surfaced in ChatGPT's search results, and OpenAI states plainly that sites opted out of OAI-SearchBot won't appear in ChatGPT search answers. ChatGPT-User is different again - a genuinely user-triggered agent for actions a person initiates directly, and OpenAI is explicit that "ChatGPT-User is not used to determine whether content may appear in Search." It's also the one OpenAI crawler with a robots.txt exception: "because these actions are initiated by a user, robots.txt rules may not apply." All three - GPTBot, OAI-SearchBot, and ChatGPT-User - were observed not executing JavaScript in Vercel's December 2024 tests; a March 2026 secondary analysis (updated June 2026), not a first-party OpenAI source, reports the same non-execution behavior more recently but without disclosing a comparable test design, so treat it as corroboration rather than an independent confirmation carrying equal weight.

ClaudeBot (Anthropic)

ClaudeBot ran a distant second by volume in Vercel's December 2024 data - 370 million monthly requests across Vercel's network - and on nextjs.org specifically it skewed toward images rather than HTML (35.17% of its fetches there) and fetched JavaScript at a notably higher rate than GPTBot (23.84% of requests, per Vercel), without executing it in that test. The more current picture is a decline, not a plateau: Cloudflare found ClaudeBot's request volume down 46% from May 2024 to May 2025 - its share of Cloudflare's broader crawler cohort fell from 11.7% to 5.4% over the same period, and its share of the AI-only cohort fell from 27% to 21% - while GPTBot surged past it. That's a real trend in Cloudflare's aggregate cohort, not a claim about any specific site's own logs; whether it changes your own fix priority depends on what your own traffic actually shows, not this aggregate share (see the prioritization section below). Claude-SearchBot is ClaudeBot's separate search-indexing counterpart - Anthropic describes it as analyzing content "to enhance the relevance and accuracy of search responses," and says disabling it "may reduce your site's visibility and accuracy in user search results." Claude-User is the third, genuinely user-triggered agent - Anthropic's own documentation says it "may access websites" when "individuals ask questions to Claude." Notably, Anthropic states all three ClaudeBot-family agents, including Claude-User, honor standard robots.txt directives - unlike every other operator on this page, it makes no explicit user-triggered exception.

PerplexityBot

PerplexityBot itself is Perplexity's search-indexing crawler, not a live-per-query fetcher - Perplexity's own documentation describes it as "designed to surface and link websites in search results on Perplexity," recommending sites allow it "to ensure your site appears in search results." The agent that actually visits a page in response to a specific question is a separate one, Perplexity-User: Perplexity says it "might visit a web page to help provide an accurate answer" when "users ask Perplexity a question" - and, unlike PerplexityBot, Perplexity states this fetcher "generally ignores robots.txt rules" since the fetch was user-requested. PerplexityBot's absolute volume was far lower than GPTBot's or ClaudeBot's in Vercel's December 2024 data (24.4 million monthly requests across Vercel's network), but Cloudflare's growth data tells a more dramatic story: PerplexityBot's request volume grew 157,490% from May 2024 to May 2025 - off a small base (under 0.01% to 0.2% of the broader cohort), but the steepest growth curve of any major AI crawler in that window. The underlying tested behavior hasn't changed: no JavaScript execution in Vercel's test. Neither Vercel's nor Cloudflare's public data breaks out PerplexityBot's 404 rate or content-type split the way it does for GPTBot and ClaudeBot, and neither publishes equivalent volume figures for Perplexity-User, so those comparisons aren't available to cite honestly yet.

Also worth knowing: Bytespider (ByteDance/TikTok's crawler) held 42% of the AI-only cohort in May 2024 - the largest share in that dataset - then collapsed 85% in raw requests by May 2025, down to 7.2% of that same cohort. Measured against Cloudflare's broader cohort including search bots, its rank fell hard too: from 2nd to 8th place, now just 2.9% of that combined traffic. Whichever cohort you look at, this was a real collapse, not a plateau - and if a "biggest AI crawler" ranking is more than a few months old, don't trust it without a date attached.

The Googlebot and AppleBot Benchmark

Googlebot dwarfed every AI crawler combined in Vercel's snapshot - 4.5 billion monthly requests across Vercel's network - and it renders JavaScript through a real headless Chrome instance, with an 8.22% 404 rate roughly a quarter of GPTBot's or ClaudeBot's. Cloudflare's more recent data shows Googlebot's own share of that broader crawler cohort still growing (30% to 50%, +96% raw requests from May 2024 to May 2025) even as it already dominated; Cloudflare separately notes crawling traffic peaked even higher in April 2025, 145% above the May 2024 baseline. AppleBot, less discussed in this context, also renders content in a browser context at meaningful scale (314 million monthly requests in the Vercel data) - though Apple's own documentation caveats that Applebot "may" fail to render properly if JavaScript, CSS, or other resources it needs are blocked, so "renders" isn't an unconditional guarantee even for Apple's own crawler. Taken together, Googlebot and Applebot demonstrate that large-scale JavaScript execution during crawling is technically feasible - not that every page gets fully rendered without timing, resource, or access constraints, and not that GPTBot, ClaudeBot, or PerplexityBot skipping it reflects some technical limitation rather than a design choice each provider has made so far.

What This Means for Prioritization

Cloudflare's aggregate cohort shares are useful background, not an allocation rule for your own engineering time. A low-volume agent can matter enormously to one site's actual buyers while a high-volume training crawler carries no immediate business priority for that same site - and the reverse is just as true elsewhere. Prioritize using your own verified (not spoofed-claiming) traffic logs, which platforms your actual audience uses, the business value of the affected pages, and - the one thing that's true regardless of which bot you're optimizing for - whether the content that matters is present in the initial HTML response rather than only added after the fact.

With that caveat, here's a reasonable default framing using three operational categories rather than a training/retrieval binary: model-development crawlers (GPTBot, ClaudeBot) collect content that may contribute to a future model, with no disclosed timeline for when or whether that happens; search/index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) crawl ahead of time to stay eligible for a platform's search results - allowing them preserves eligibility, but none of the three providers promises indexing, selection, or citation as a result; and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch a page at the moment someone asks about it directly. If a specific answer citing your page right now is the priority, the search/index and user-triggered categories are the more directly relevant paths to check first - but that's a judgment call about what matters to your business, not a volume-derived fact, and Cloudflare's own data shows training-purpose crawling still accounts for most AI-bot traffic by volume (nearly 80% in its August 2025 analysis). Either way, the underlying fix for the crawlers that don't execute JavaScript is the same rendering work - see the diagnostic and remediation pages below.

← Back to the full pillar page
Living AI crawler user-agent reference →
Test which of these bots can actually read your site →

---

Sources: December 2024 volume figures (network-wide) and JS-fetch/content-type/404-rate figures (nextjs.org-specific, per Vercel's own scoping, cross-checked by Vercel against two additional job-board sites) from Vercel, "The rise of the AI crawler" - dated deliberately in every reference above rather than implied current. Growth and share trends (May 2024 to May 2025), including both the AI-only and broader-cohort tables, from Cloudflare, "From Googlebot to GPTBot: Who's crawling your site in 2025" (published July 2025). Four-category traffic-by-purpose split (Training/Search/User action/Undeclared) from Cloudflare, "A deeper look at AI crawlers: breaking down traffic by purpose and industry" (August 28, 2025) - the nearly-80%/under-5% figures are quoted directly; the "rest is Search" framing is our own arithmetic on those two figures, not a Cloudflare quote. Crawler-vs-fetcher purpose distinctions and exact language for GPTBot/OAI-SearchBot/ChatGPT-User from OpenAI's official bot documentation; for ClaudeBot/Claude-SearchBot/Claude-User from Anthropic's own crawler documentation; for PerplexityBot/Perplexity-User from Perplexity's official crawler documentation. Applebot's conditional rendering language from Apple's official Applebot documentation. A secondary, more recent non-execution observation for several named agents including Claude-SearchBot comes from Lantern's crawler analysis (originally published March 2026, updated June 2026) - treated as corroboration, not a substitute for Vercel's disclosed test, since Lantern doesn't publish its own methodology or sample. This page will need revisiting again within a few months - this data category moves fast.