How to Test If AI Bots Can Actually Read Your JS Content

Most advice on this stops at "open DevTools, disable JavaScript, see what's left." That's a fine five-second sanity check, but it doesn't tell you what's actually happening in production - whether GPTBot, ClaudeBot, and PerplexityBot are actually hitting your site, getting through, and receiving a page with your content in it. Getting a real answer means triangulating three separate questions - did a verified bot show up, did it get a usable response, and was the specific content actually in that response - not treating any single signal as proof by itself. It takes about fifteen minutes the first time.

The Quick Sanity Check (View-Source)

Right-click any page and choose "View Page Source" - not "Inspect," which shows the post-JavaScript DOM. If your pricing, features, or docs content isn't visible in that raw source, it won't be visible to a crawler that doesn't execute JavaScript. In Vercel's published December 2024 tests, that described GPTBot, ClaudeBot, and PerplexityBot specifically - a dated, repeated observation about those three named agents, not a permanent guarantee any of the three providers has made, so it's worth re-testing periodically rather than treating it as settled forever. This ten-second check catches the obvious cases. It won't tell you whether these bots are actually visiting, how often, or whether something more subtle - a redirect, a bot-specific block, a timeout, response variance by client - is getting in the way. For that, you need the logs, and then a body-level content check, not either one alone.

The Real Test: Your Server Logs

Step 1 - Pull your access logs

Wherever your logs live - Nginx or Apache access logs, your CDN's raw logs (Cloudflare, Fastly, CloudFront), or your hosting platform's log export - pull a recent window. A week is usually enough to see a pattern; a single day can miss lower-volume bots like PerplexityBot.

Step 2 - Filter for AI bot traffic

Filter for the identifying substring each bot uses in its user-agent string. A standard access log line, filtered with grep, looks like this:

The exact full user-agent strings, where confirmed directly from the bot operators' own documentation: GPTBot uses GPTBot/1.4, ChatGPT-User uses ChatGPT-User/1.0, and OAI-SearchBot uses OAI-SearchBot/1.4 as the identifying token (version numbers change over time - match on the name, not the version). PerplexityBot uses PerplexityBot/1.0 and Perplexity-User uses Perplexity-User/1.0. Anthropic's three bots - ClaudeBot, Claude-User, Claude-SearchBot - use those names as the matching substring in robots.txt and in logs; adapt the grep pattern above to your actual log format (JSON logs, combined log format, and CDN-specific formats all need slightly different syntax).

Step 3 - Look at what they hit, and what they got back

Pull the URL path and HTTP status code for each matched line. Watch for bots hitting 404s on pages that exist (a redirect chain, a broken sitemap, or a bot-specific block worth investigating) and for anything other than a clean 200 - a 403, a 429, a 5xx, or a challenge page all mean the request never reached usable content, which is a different problem from a rendering gap. Response sizes are worth glancing at here too, but treat them as a prioritization signal, not a finding: if GPTBot's hits on your pricing page and your empty error page come back at suspiciously similar byte sizes, that's worth checking more closely, not a confirmed diagnosis on its own. Similar byte counts don't prove identical content, and different byte counts can come from compression, response headers, personalization, caching, or an error template rather than an actual content gap - and a log line only ever tells you the status and byte count that were logged, never the body itself, since standard access logs (NGINX's default format, Cloudflare's request logs) record request/response metadata, not response content.

Step 4 - Confirm what's actually in the response body

This is the step that actually settles it - everything before this point narrows down where to look. Response size is a heuristic; the presence or absence of your actual content in the raw response body is the pass criterion. Be precise about what a plain curl request does and doesn't prove: curl sends its own default user agent (curl/ plus its version) and, by default, does not follow HTTP redirects - you have to tell it to. And the representation a server returns can vary by request headers, cookies, cache state, source IP, geography, or WAF/bot-allowlist rules, so a plain curl request from your own machine is a genuine no-JavaScript content check, not a guaranteed replay of what a specific historical GPTBot, ClaudeBot, or PerplexityBot request actually received. Run it accordingly:

-L follows redirects instead of silently stopping at a 3xx; --compressed and saving headers separately let you check the final status, Content-Type, Content-Encoding, and any Vary/cache/WAF headers before you interpret an empty grep result as a content gap rather than a delivery quirk. If you want to test whether a site behaves differently for a specific bot's user agent, you can add -A with that operator's current documented string - but label that run for what it is: a simulation from your own IP with a spoofed header, not a verified fetch from the provider's actual infrastructure. It can surface UA-conditional behavior; it can't reproduce the provider's real network origin, cookies, or cache treatment. If the final response body genuinely lacks the text that a browser clearly shows, you've confirmed a real content gap in the initial HTML - not proof of what any specific historical bot request received, but strong, directly-inspected evidence of what the page currently serves.

Step 5 - Verify it's really the bot, not a spoofer

User-agent strings can be faked by anyone - scrapers spoof "GPTBot" constantly to get past bot-specific allowlists. Before you make decisions based on log volume, cross-reference the source IPs against each operator's published list: OpenAI publishes IP ranges at openai.com/gptbot.json, openai.com/chatgpt-user.json, and openai.com/searchbot.json; Anthropic publishes a combined list at claude.com/crawling/bots.json, and states directly that a source IP on that list "indicates that the crawler is coming from Anthropic"; Perplexity publishes theirs at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. If a "GPTBot" hit in your logs isn't coming from an IP on OpenAI's list, it's not GPTBot - and it shouldn't factor into any decision about what these crawlers can or can't see.

One thing this check depends on that's easy to get wrong: if your site sits behind a CDN or reverse proxy (Cloudflare, CloudFront, Fastly, a load balancer), your origin logs may record the proxy's own address, not the original client's - unless the real-client-IP field (an X-Forwarded-For or provider-specific header) is configured and your proxy only trusts that field from intermediaries you actually trust. Comparing the wrong field to a provider's published range will either reject legitimate bot traffic or validate an address that isn't actually the bot's. Confirm which field carries the real client IP in your logging setup before treating a match - or a non-match - as conclusive; a non-match by itself means the request is unverified, which could mean spoofed, mis-logged, proxied incorrectly, or checked against a since-changed list, not automatically a specific cause. Provider IP lists also change over time, so pull the current list rather than a saved copy.

Reading Your Results

A single pass/fail label blurs together questions that have different causes and different fixes - "no bot ever showed up," "the bot showed up but got blocked," and "the bot got through but the content wasn't there" are three different problems. Instead of one binary result, place what you found into one of these:

Not observed: no verified (IP-confirmed) request from the bot appears in your log window. Extend the window before concluding anything - a week is usually enough, but low-volume bots like PerplexityBot can take longer to show up. Don't infer readability or blocking from an absence of visits; that's a separate question this test doesn't answer.

Access blocked or failed: a verified request received a 401, 403, 429, 5xx, a bot challenge, a timeout, or a broken redirect chain. This is a delivery/access problem - robots.txt, WAF rules, routing, rate limiting, or infrastructure - not a rendering problem, and it needs a different fix than anything in this cluster's SSR/prerendering guidance.

Reached, content absent: a verified request got a successful final HTML response (after following any redirects), but Step 4's body inspection didn't find the critical content marker. This is the case that points toward client-only rendering or data fetching - but confirm it's not something else first: response variance by client, personalization, consent/cookie state, geolocation, or a genuinely broken or incorrect page can all produce the same symptom.

Reached, content present: the critical marker exists in the initial response body a verified bot would receive. This passes the initial-HTML check specifically - it removes one access/rendering obstacle. It does not by itself establish indexing, retrieval, or citation; those are separate, later decisions each platform makes on its own.

Unexpected 404: before assuming a block, verify the exact URL requested and where any redirect chain actually lands - a stale sitemap entry, a bad internal link, or bot-specific routing can all produce a 404 that has nothing to do with rendering.

What to Do Next

Don't route every "content absent" result straight to a rendering fix - that's the one mistake most likely to waste engineering time on the wrong problem. If a verified request reaches a successful final HTML response and the specific content is missing there while a human in a browser only sees it after JavaScript runs, client-only rendering or data fetching is the leading diagnosis, and the pages below are the right next step. But if what you actually found was blocked/failed access, or status/redirects/headers/cache/body that vary by client, resolve that delivery issue first - a rendering fix won't touch a 403 or a misconfigured redirect. Only move on to the SSR/prerendering guidance once you've confirmed, via Step 4's body inspection, that the content is genuinely missing from the relevant initial HTML response and not blocked, redirected, or served differently for a reason unrelated to rendering.

← Back to the full pillar page
Why this happens: hydration and framework-specific failure modes →
Fixing it: SSR vs. prerendering decision guide →

---

Sources: User-agent strings and IP-verification endpoints for GPTBot, ChatGPT-User, and OAI-SearchBot from OpenAI's official bot documentation. PerplexityBot and Perplexity-User strings and endpoints from Perplexity's official crawler documentation. Anthropic's bot names and IP-verification endpoint, including the exact "indicates that the crawler is coming from Anthropic" language, from Anthropic's own crawler documentation - the exact cosmetic user-agent prefix for Anthropic's three bots wasn't independently confirmed and isn't claimed here, only the identifying name each uses for matching. Non-execution behavior for GPTBot, ClaudeBot, and PerplexityBot specifically (December 2024, dated and agent-scoped) from Vercel's crawler study. curl's default user agent and default no-redirect-follow behavior from curl's own HTTP scripting documentation. Request-dependent response representation (why a plain fetch isn't a guaranteed replay of a specific historical request) from RFC 9110's content-negotiation and status-code sections. What standard access logs do and don't record from NGINX's access-log module documentation and Cloudflare's logging documentation - both document request/response metadata, not response body content.