How AI Crawlers Actually Access JS-Heavy SaaS Sites

GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript. If your SaaS site renders its content client-side - React, Vue, Next.js without server-side rendering, whatever the framework - these bots see whatever's in the raw HTML response and nothing else. The clearest direct evidence for that specific claim is Vercel's crawler-traffic analysis - worth flagging honestly that it's dated December 2024, and we haven't found a fresher primary study that directly re-tested JS execution itself. Every SaaS team shipping JS-heavy marketing pages still needs to plan around this.

Here's where it gets more useful than that one fact: not every AI crawler is doing the same job, and treating them as one bucket is why a lot of "fix your AI crawlability" advice doesn't actually tell you what to fix first.

The Distinction Sites Need to Make: Training, Retrieval, and User-Triggered Fetch

Three purposes, not two. Training crawlers - GPTBot, ClaudeBot, Bytespider - harvest content to build or update a model; allowing one makes your public content eligible for potential future model improvement, subject to that provider's own selection, filtering, training, and release process. No provider publishes a guaranteed timeline for when, or whether, a given page actually ends up shaped into a model, so treat "it'll show up in training eventually" as an eligibility statement, not a schedule. Retrieval crawlers - OAI-SearchBot, Claude-SearchBot, PerplexityBot - build or refresh a search index the answer engine draws on; they run on their own schedule, not at the instant someone asks a question. User-triggered fetchers - ChatGPT-User, Claude-User, Perplexity-User - are the ones that actually visit a page live, the moment a user asks that specific product to check it; providers note that robots.txt rules may not fully apply to this category the way they do to the other two. Most "AI crawler" advice collapses the last two into one bucket, which is exactly backwards for figuring out what a failed request actually costs you.

The difference matters for prioritization, but more carefully than "fix retrieval first." Miss a training crawler and, at most, your site's representation in a future model lags reality - with no guaranteed timeline either way. Block a retrieval crawler and your pages may become less eligible for discovery or slower to refresh in an engine's index. Block a user-triggered fetcher and a specific user's specific request may fail outright. None of that guarantees a citation, a ranking, or a training outcome on its own - crawlability is necessary for reliable access, not sufficient for any particular result. If you can only prioritize one path, prioritize whichever category your logs show is actually reaching your key pages least reliably, not a fixed default answer.

Training crawling is the larger share of what's actually hitting a site, not the smaller one: in Cloudflare's own four-category breakdown of AI bot traffic (Training, Search, User action, Undeclared), training-purpose traffic runs nearly 80% of the total, with Search accounting for most of the rest - User action and Undeclared traffic combined run under 5%. That's a genuinely small slice of live, user-triggered fetching relative to everything else hitting a site, which is itself worth knowing: most "AI crawler" traffic isn't answering anyone's question in real time. And the landscape moves fast enough that any single snapshot goes stale within months: Cloudflare found GPTBot's share of its broader crawler cohort (AI and search bots combined) grow from 2.2% to 7.7% in the year to May 2025 - a 305% jump in raw requests - while ClaudeBot's requests fell 46% (11.7% to 5.4% of that cohort) and Bytespider fell from 2nd to 8th place in the same ranking, collapsing 85% in requests down to just a 2.9% share. PerplexityBot grew 157,490% off a small base over the same period. None of that changes the JS-rendering conclusion, but "which bot matters most" is a moving target, not a fact you fix once.

See the full bot-by-bot comparison →
Living AI crawler user-agent reference →

Why JS-Heavy SaaS Sites Break This

A client-side-rendered SaaS site sends a near-empty HTML shell on the first response - a single container element and a pile of script tags - and fills in the actual content in the browser, after JavaScript runs. That works fine for a human visitor. It fails completely for a crawler that fetches the HTML and stops there, which is exactly what GPTBot, ClaudeBot, and PerplexityBot do: they'll download your JavaScript files (ClaudeBot pulls them in roughly 24% of its requests) without ever executing them.

The practical result: your pricing page, your feature comparison, your docs - anything whose meaningful content only exists after client-side JavaScript runs, because it's rendered or fetched in after the initial response - can be functionally blank to these bots, even though it looks completely normal in a browser. Worth being precise about the mechanism here: hydration itself isn't the problem. Hydration is React (or a similar framework) attaching interactivity to HTML the server already sent - if the server response already contains the meaningful content, a crawler that never runs your JavaScript still sees that content just fine. The actual failure mode is client-only rendering, or content fetched and inserted after the page has already loaded.

Framework-specific failure modes: React, Vue, Next.js, Nuxt →

The Exceptions: Googlebot, Gemini, and AppleBot

Googlebot renders JavaScript through a documented crawl-render-index pipeline - it isn't a new capability from 2019; Google's crawler could already render pages earlier using an older, fixed Chrome build. What changed in 2019 specifically was the switch to "evergreen" rendering - Google keeps Googlebot's rendering engine continuously updated to the current stable Chromium, rather than leaving it frozen on an aging version. Because Gemini's web results draw on that same infrastructure, it's one AI-relevant surface where client-side rendering doesn't automatically break your visibility. Apple's own documentation says Applebot "may render the content of your website within a browser" - a real capability, not a guarantee: Apple's own caveat is that blocked JavaScript, CSS, or XHR resources can prevent complete rendering, so don't treat Applebot as rendering every page fully by default any more than you'd assume that for a page with broken resource loading. Vercel's data shows Applebot fetching JavaScript at real scale in its own dataset (314 million monthly requests) - so this isn't purely a Google-infrastructure story, but it's also not an unconditional guarantee. Every AI crawler this page has covered that isn't in this exceptions section - GPTBot, ClaudeBot, PerplexityBot, and the retrieval and user-triggered agents tested by Vercel in December 2024 - did not execute JavaScript in that study; still treat your site as invisible to them by default until you've tested it yourself.

How to Check Your Own Site

View-source is a useful initial-response check, worth doing as a first pass - but it only tells you what your own browser requested, not what a bot actually received. And server access logs alone won't settle it either: ordinary logs record requests and metadata - user agent, status code, byte count, timing - they don't normally store the response body, so a log full of clean 200s doesn't prove a crawler got your full content instead of an empty shell. Use logs to confirm which verified agents requested which URLs and how big the response was; then fetch the raw HTTP response yourself with the relevant user agent, inspect it directly for the headings, links, and structured data that actually matter, and compare that against what renders in a browser's DOM. Verify the traffic is genuine with each operator's published IP ranges before drawing conclusions from any of it - a user-agent string alone is just a text field anyone can send.

The full log-based self-test, step by step →

Fixing It: SSR, Prerendering, and the Real Tradeoff

Framework choice isn't the deciding factor by itself - the test is whether critical content actually ships in the initial HTML response, however that happens. Full server-side rendering is one reliable way to get there, and frameworks like Next.js, Nuxt, and Angular Universal support it - but Next.js specifically can also ship content-complete initial output through static generation, incremental static regeneration, or server components, without every request going through request-time SSR; "Next.js without SSR" isn't automatically invisible the way plain client-only React can be. Static generation pre-builds pages at deploy time and works well for content that doesn't change per request. Dynamic rendering - serving bots a separately generated snapshot while humans get the normal client-rendered app, via middleware like Prerender.io - is explicitly described by Google's own documentation as a workaround, not a recommended long-term solution, because of the added complexity and resource requirements it introduces; it's a legitimate temporary bridge for a legacy client-only app you can't rearchitect this quarter, not the first choice. (Google's own earlier reference implementation for this approach, Rendertron, was archived and explicitly deprecated back in October 2022 - don't take advice that still points to it.) A snapshot served this way also has to stay materially equivalent to what a real user sees, or it risks looking like deceptive serving rather than a legitimate accessibility bridge. Every approach here carries engineering, infrastructure, vendor, or maintenance cost - the right one depends on how much of your build pipeline you're willing to touch right now, and prefer server-rendered or statically generated content where it's practical rather than defaulting to a snapshot service.

The full SSR vs. prerendering decision guide →
Implementation checklist →

What About llms.txt?

No major AI provider - not OpenAI, not Anthropic, not Perplexity - has publicly committed to honoring llms.txt as of mid-2026. It costs almost nothing to add and it won't hurt you, but if your engineering backlog is already full, it isn't the ticket to prioritize this quarter.

The full llms.txt verdict →

AI Crawlers Are Inefficient Visitors - Worth Cleaning Up, Not a Proven Quota

In Vercel's December 2024 dataset, ChatGPT's crawlers hit 404 pages on 34.82% of requests and Claude's on 34.16% - compare that to Googlebot's 8.22% in the same study. A meaningful chunk of those misses were requests for obsolete static assets sitting in old /static/ paths, not broken links to real content. Treat that as evidence of genuine crawl inefficiency and noisy logs worth cleaning up - stale sitemaps, orphaned redirect chains, an exposed staging subdomain - not as proof of a fixed, Google-style per-site "AI crawl budget" that bad URLs are eating into. That mechanism sounds plausible, but it isn't something Vercel's study, or any other source used in this cluster, actually tested; these figures are also specific to Vercel's December 2024 dataset, not an established universal rate for every site.

AI crawl budget vs. SEO crawl budget →

What to Watch

The following is analysis and opinion from Zarko Zivkovic, not established fact - flagged as such deliberately.

Agentic AI browsers - the kind that open a real rendered browser and click around the way a person would - already exist as shipping products, not a someday technology. When one of these agents visits a specific page at a user's direction, it renders JavaScript like a real browser would, and the client-side-rendering gap this page describes genuinely doesn't apply to that single visit. What's still true: browser-execution agents are not the default, scalable discovery and indexing path for any major answer engine today - running a full rendered browser session per candidate page during discovery would be far more expensive than the lightweight HTML fetch that retrieval crawlers already use, and none of the major agentic products are positioned as a replacement for that discovery layer. An agent solving the problem at the point of one direct visit doesn't mean a page gets found in the first place - it still has to get discovered through the same crawler infrastructure this whole page covers. Crawlable server output remains the more dependable baseline for discovery, retrieval, latency, and cost, agentic browsers or not; don't deprioritize a rendering fix on the assumption that "an agent will read it anyway."

Full analysis: agentic browsers vs. crawling →

---

Sources: Core JS-non-execution finding, and the 404-rate figures, from Vercel, "The rise of the AI crawler" (December 2024 - the most direct primary test found; no fresher primary re-test of JS execution was located; figures are specific to Vercel's own dataset and the agents it tested, not a universal rate). Crawler purpose and naming - training, search/retrieval, and user-triggered fetch - are from each operator's own current documentation: OpenAI's crawler documentation, Anthropic's crawler documentation, and Perplexity's crawler documentation - see the user-agent reference page for the full current list. Growth, share, and ranking trends from Cloudflare, "From Googlebot to GPTBot: Who's crawling your site in 2025" (July 2025, data covering May 2024 to May 2025) - figures above use Cloudflare's broader 30+ crawler cohort table, not its narrower AI-only-crawler table, which shows different (larger) share percentages for the same bots. The four-category traffic-purpose split (Training/Search/User action/Undeclared) is from Cloudflare, "A deeper look at AI crawlers: breaking down traffic by purpose and industry" (August 28, 2025), quoted directly - "retrieval" is this page's own descriptive term for the Search category, not Cloudflare's. Googlebot's rendering pipeline and the 2019 evergreen-Chromium timeline are from Google's JavaScript SEO basics documentation and Google's 2019 evergreen Googlebot announcement. Applebot's rendering caveats are from Apple's own Applebot documentation. Dynamic rendering's status as a workaround, not a recommended long-term solution, is from Google's dynamic rendering guidance. Rendertron's archived/deprecated status is documented at github.com/GoogleChrome/rendertron, archived October 6, 2022. Hydration's definition is from React's own hydrateRoot documentation; Next.js's multiple rendering strategies are documented at nextjs.org. This page will be updated as crawler behavior and llms.txt adoption change - both are moving targets in mid-2026.