How do I check whether ChatGPT and other AI engines can actually crawl my website?

Open yoursite.com/robots.txt in a browser or fetch it with curl -I https://yoursite.com/robots.txt to confirm it returns a 200 and not a redirect chain that some CDNs insert. Then read the file for explicit Allow or Disallow blocks naming each of these agents by their exact token: GPTBot and OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), ClaudeBot (Anthropic), Google-Extended (Google's AI training and Gemini grounding signal, separate from Googlebot), Applebot-Extended (Apple Intelligence), and Bingbot (which feeds both Bing and, indirectly, ChatGPT's Bing-derived retrieval index).

The trap almost everyone falls into: a robots.txt with no mention of GPTBot at all is not the same as a robots.txt that allows GPTBot. Absence of a Disallow line is technically permissive under the robots exclusion protocol, but plenty of site owners assume silence means blocked and never verify, while others assume silence means allowed and are wrong because a wildcard rule higher in the file (User-agent: * followed by Disallow: /) already caught the bot before it ever reached a bot-specific block. Read every User-agent block in the file, not just the ones with a familiar name, and check which one actually applies to each crawler using first-match-wins logic per section.

Robots.txt is a request, not an enforcement mechanism, and it only matters if the crawler bothers to read it and complies. The bigger real-world blocker is infrastructure-level filtering: Cloudflare has shipped default rulesets on some plans that block or challenge known AI-crawler user agents at the edge, before your robots.txt or even your server is consulted, and Cloudflare's own AI-bot-blocking toggle is on by default for some account types created after mid-2024. Log into your Cloudflare dashboard (or whatever CDN or WAF you run) and check Security > Bots for any rule referencing "AI crawlers," "AI Scrapers and Crawlers," or a managed rule with GPTBot, ClaudeBot, or CCBot named explicitly, and confirm it is set to Allow rather than Block or JS Challenge.

The same check applies to any bot-management product (Akamai, Imperva, Fastly, DataDome) sitting in front of your origin, and to a generic rate limiter that treats a burst of requests from a single AI-crawler IP range as abuse and starts serving 403s.

Run curl -A "GPTBot" -s https://yoursite.com/your-page > gptbot.html and, separately, curl -A "Mozilla/5.0" -s https://yoursite.com/your-page > browser.html, then diff the two files or just open gptbot.html in a text editor and search for your key content: your services list, your address, your main value proposition. If gptbot.html is dramatically shorter, or a WAF challenge page, or a blank shell, something in the chain is blocking or throttling that agent even though robots.txt looked fine.

Repeat with -A "ClaudeBot" and -A "PerplexityBot" since each vendor's infrastructure treats agents differently, and a rule blocking one does not necessarily block the others.

Most AI crawlers, including GPTBot and ClaudeBot, fetch raw HTML and do not execute JavaScript the way a browser or Googlebot's renderer does. If your services, pricing, or location content is injected client-side by a React, Vue, or Angular app after the initial page load, the curl output above will show an empty div or a loading skeleton where your actual content should be, even though the page looks completely normal in a browser.

View the raw source with curl -s https://yoursite.com/ | grep -i "your service name" rather than trusting the browser's rendered DOM or even View Page Source in some browsers, which can show post-hydration markup depending on settings. If your content only appears after JavaScript runs, you need server-side rendering, static generation, or a prerendering service in front of the site, because an AI crawler that never sees your text cannot cite your business regardless of how well the site converts human visitors.

Single-page applications commonly configure their server or CDN to return index.html with a 200 status for any unmatched path, so the app can handle client-side routing. This creates a specific false positive: if you test a URL that does not actually exist, such as a deleted blog post or a mistyped path, curl will return HTTP 200 with Content-Type: text/html and a full page of markup, and it will look like a successful crawl even though the specific page you intended to check was never served.

Always verify the status code and the actual content, not just the 200. Run curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/path-you-are-testing and then separately confirm the body actually contains content specific to that page (grep for its H1 or a unique product name) rather than your generic homepage shell or a client-side "page not found" component that a raw HTML fetch cannot see because it only renders after JavaScript runs.

Robots.txt and curl tests confirm permission and capability; server logs confirm it actually happened. Pull your raw access logs (via your host's control panel, a logging add-on, or SSH if you manage the server directly) and grep for the exact bot tokens: grep -i "GPTBot" access.log, grep -i "ClaudeBot" access.log, grep -i "PerplexityBot" access.log, grep -i "Google-Extended" access.log. Look at request frequency, which paths were hit, and what status codes were returned, since a bot that visited once a month ago and got a string of 403s or 500s is effectively not crawling your site even though it tried.

If your logs show zero hits from any of these agents over a 30-day window, the problem is upstream of your server: either something is blocking them before they reach you, or your site has not yet been discovered because it lacks the sitemap submission, backlinks, or crawl budget signals that would put it in the relevant crawl queue.