Docs

Bot Failures & Investigations

Why does a bot show “Failure” or “Needs Investigated”?

If a bot in the AI Search Console shows “Failure” or “Needs Investigated,” it means our simulated test could not confirm that the bot can access your site.

However, your site might still be indexed successfully — sometimes the block only affects the test. The steps below will usually tell you exactly what’s blocking the bots and who can fix it, in about ten minutes.

Step 1: Is it one bot, or everything?

This is the most useful question to answer first.

  • One or two bots failing usually means a targeted rule — a robots.txt entry or a firewall rule aimed at that specific bot.
  • Every bot failing usually means a bot-management or firewall layer in front of your site that challenges everything that isn’t a real person in a real browser — including our tests.

To check, fetch your site with a tool that isn’t a browser:

  • Browser-based: paste your URL into a checker like httpstatus.io. If it reports a 403 (or another error) while the site loads fine in your own browser, a protection layer is challenging non-browser visitors.

  • Terminal (Mac/Linux):

    curl -I https://www.yoursite.com/ curl -I -A "GPTBot/1.0 (+https://openai.com/gptbot)" https://www.yoursite.com/

    If both commands return 403, the block isn’t about AI bots specifically — everything non-browser is being challenged. If only the GPTBot one fails, the rule targets bots by name.

Step 2: Find out who is doing the blocking

The 403 error page usually signs its own work. Look at the page body (it often says “Cloudflare,” “Sucuri,” or your host’s name) and, if you used curl -I, the response headers:

  • server: cloudflare plus cf-mitigated: challenge — Cloudflare is serving a challenge page instead of your site. Humans pass the challenge invisibly; bots and test tools can’t.
  • x-amz-cf-id — Amazon CloudFront
  • x-sucuri-id — Sucuri firewall
  • x-served-by — Fastly

A challenge page is the classic “site is up for me, but every bot shows Failure” cause.

Step 3: Find out who actually hosts your site

Many business sites run on a website platform behind the scenes — RentCafe, Wix, Squarespace, Shopify, and similar. These platforms often run bot protection platform-wide, which means nobody on your team turned it on, and no setting on your site can turn it off. The fix is a support request to the platform, not to your IT person or web designer.

To check, look up your domain’s DNS at dnschecker.org (CNAME lookup), or in a terminal:

nslookup www.yoursite.com

If the answer shows a chain through another company’s domain (for example www-yoursite-com.rentcafecn.com), that company hosts your site — and their bot protection applies to it.

Step 4: Check your robots.txt

Open https://www.yoursite.com/robots.txt in your browser and look for Disallow rules under User-agent: lines naming AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.). This is a separate mechanism from a firewall block: robots.txt politely asks bots to stay away, and well-behaved AI crawlers comply.

Step 5: If you control Cloudflare

If your site is on Cloudflare and you have access to the dashboard:

  • Security → Bots — check whether Bot Fight Mode or Super Bot Fight Mode is on, and whether the block AI bots / AI crawlers setting is enabled. Cloudflare turns AI-crawler blocking on by default for newly added domains, so you may be blocking AI bots without ever having chosen to.
  • Security → Events — the log shows each blocked or challenged request and which rule fired, so you can see exactly what happened to the bot traffic.

Why our test can fail when real bots succeed

We test by sending requests with each bot’s User-Agent string, but the requests come from our servers — we can’t send them from OpenAI’s or Google’s actual IP addresses. Security services that verify bots by their source network may challenge our test while letting the genuine crawlers through. See What does “403” mean in the AI Search Console?

Two ways to check what’s really happening:

  • Server logs: look for recent visits from the real bots. The vendors publish their crawler IP ranges, so your host or IT team can confirm whether genuine GPTBot or ClaudeBot traffic is getting through.
  • Branded search: the end-to-end proof — see Persistent Console Failures.

If you contact support

Include these and we can usually skip a round of back-and-forth:

  • Your domain, and whether one bot fails or all of them do (Step 1)
  • Anything the 403 page or headers revealed (Step 2)
  • Your hosting platform, if you found one (Step 3)
  • What you’ve already tried or ruled out