Botglass: what each AI bot actually receives from your URL

Because the request did not come from Google. Botglass sends Googlebot’s User-Agent from its own IP address, which is exactly what a spoofed crawler looks like, and bot-management systems commonly check Googlebot by IP range or reverse DNS. A challenge here often means your firewall is working. It does not show that the real Googlebot is blocked. To check that, use the live test in Google Search Console’s URL Inspection tool, and look in your server logs for Googlebot requests verified against Google’s published IP ranges.

Because the refusal matched the stock response of Cloudflare’s managed “Block AI bots” setting: HTTP 403, a plain-text body reading “Your request was blocked.”, served at the edge within milliseconds, and no robots.txt rule involved. That is a fingerprint, not a reading of your Cloudflare settings, so Botglass names the setting as a likely cause rather than a certainty. It has been on by default for new Cloudflare zones since July 2025, and older zones are offered it as a one-click prompt.

If you want the site to appear in AI answers without feeding model training, Cloudflare’s AI Crawl Control lets you allow the search and answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot) while keeping GPTBot, CCBot and Bytespider blocked. Botglass shows this split as “training” versus “visibility” in the verdict panel, so the cost of a block is clear. Run Botglass again afterwards to confirm the change.

There is no documented link. GPTBot is OpenAI’s training crawler: disallowing it tells OpenAI not to use your content for training. What can matter is a firewall rule broad enough to catch Googlebot too. Two related points: OAI-SearchBot is a separate OpenAI token for ChatGPT search, so blocking it changes whether OpenAI can index your pages for search; and Google-Extended governs Gemini training and grounding, and Google says it does not affect Search inclusion or ranking. Botglass shows each token separately for this reason.

Botglass reports whether the file exists; it does not claim that the file helps. llms.txt is a community proposal, not a standard, and we are not aware of a major AI vendor confirming that it reads the file when retrieving or ranking pages. It takes little time to add and does little harm if ignored, so treat it as optional housekeeping rather than a visibility lever. Some SEO plugins can generate it for you.

Your logs show what real bots did. Botglass shows what happens when one request carrying a bot’s User-Agent arrives from a different IP address. Differences are expected: firewalls may treat verified IPs differently, a bot may never have visited that page, robots.txt can be served differently by User-Agent or IP, and a single fetch is a snapshot. To see which of your log entries were genuine, compare them with each vendor’s published IP list, which is linked from every row in the tool.

Sending a crawler’s User-Agent is routine in SEO tools (Screaming Frog, for example, lets you choose one), and it is the only way to see how a server responds to that name. It is still presenting as something Botglass is not, from an address the vendors do not use, so the tool is restrained: it requests only the URL you give it plus robots.txt and llms.txt, does not crawl, spaces requests out, refuses private addresses, and caps checks per visitor, per site and per day. It adds no identifying token and does not try to get past challenges. Test sites you own or have permission to test.

Results are held in memory for up to an hour so that repeat checks are quick. Application logs record the target host, not the full URL, and a salted hash of the visitor’s IP address, not the address itself. Our hosting provider and, where switched on, Cloudflare Turnstile process requests under their own terms. Do not enter URLs that contain passwords or tokens.