AI search

My Site Disappeared From AI Answers. So I Built a Free Checker.

What ChatGPT, Claude, Perplexity and Google need before they can cite a site, why most site owners can't see it, and the product decisions behind the free tool I built to check.

By Roberto Iriondo · Published November 11, 2026 · Last updated November 11, 2026

A few days ago my site stopped showing up. It was gone from the Google cards it used to appear in, and ChatGPT and Perplexity stopped citing it when I asked about my own work.

The code was fine. robots.txt allowed every crawler, every page had the right tags, and the structured data passed every test. The problem was the host. On and off for about a day, our server kept failing when Cloudflare connected to it, so new visitors and crawlers got error pages (Cloudflare's 520, 522 and 525) instead of the site. People who had visited before still saw it, because the site saves a copy in their browser. Googlebot and the AI crawlers have no such copy. When they came during an outage, they got an error.

That is the part worth writing about. From the outside, everything looked right. The only way to see the problem was to ask for the page the way each crawler does and look at the answer it got. Most site owners have no easy way to do that, so I built one: the AI Visibility Checker. It is free, needs no sign-up, and gives a score out of 100 with the fixes in order.

Key takeaways

• AI assistants can only cite a page they can reach, read and tie to a known company or person. A miss on any of the three can drop you from answers.
• robots.txt is a request. Your server, firewall or bot protection decides who actually gets in, and it can turn away crawlers that robots.txt allows.
• The crawlers behind AI answers are not the training crawlers. Blocking GPTBot does not stop ChatGPT from citing you. Blocking OAI-SearchBot does.
• Most AI crawlers do not run JavaScript. If your text only appears after scripts run, they see an empty page.
• The checker is free to run by design: no AI model calls, free public data sources, results kept for an hour, and a limit per visitor.

What AI assistants need before they cite you

When someone asks ChatGPT, Claude, Perplexity or Google's AI Overviews a question, the answer is built from pages the tool can fetch and read, often at that moment. Before your page can be one of them, three things have to be true. They became the three parts of the score.

1. Reach (52 points). The page answers, it is not set to noindex, and both robots.txt and the server let in Googlebot, Bingbot and the crawlers behind ChatGPT, Claude and Perplexity answers.
2. Understand (35 points). The text is in the HTML without running JavaScript. The page has a title and description, structured data that names the company with its website, logo and profiles, a sitemap, and an llms.txt.
3. Recognized (13 points). The company has a Wikidata item that lists the site as its official website, a Wikipedia article, and pages in Common Crawl, the open web archive many AI models learn from.

Reach carries the most weight because it is a hard gate. A page nobody can fetch scores zero in every answer, however good it is.

The product decisions behind the checker

Most of the work was deciding what the score should mean. These are the decisions that shaped it.

1. Score the crawlers that cite you, not the ones that train

Each AI company runs more than one crawler. OpenAI has GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for when a person asks ChatGPT to read a page. Anthropic and Perplexity have the same split. Google-Extended controls Gemini training and does not affect Google Search.

Blocking training crawlers is a fair business choice, and many publishers make it. It does not stop AI assistants from citing you. So the checker shows the training crawlers but gives them no points. A site that blocks GPTBot and lets OAI-SearchBot in can still score 100.

2. Check what the server actually says

Most AI visibility checks stop at robots.txt. That would have missed my own problem, and it misses a common one: a firewall, a bot protection setting or a security plugin that turns crawlers away no matter what robots.txt says.

So the checker asks for your page 13 times: once as a browser and once as each of 12 crawlers, using each crawler's own user agent. It reports what each one got, and names a Cloudflare challenge when it sees one. If a browser gets the page and PerplexityBot gets a 403, you know where to look.

There is one limit, and the page says so. Some firewalls check a crawler's network address, not only its name. The checker sends each crawler's name from Cloudflare's network, so a firewall like that may treat the real crawler differently. Your server logs have the final word.

3. Read the page without JavaScript

Google runs JavaScript before it indexes a page. Most AI crawlers do not. A site built as a JavaScript app can look perfect in a browser and send crawlers an empty <div id="root">.

The checker counts the words in the HTML as sent, before any script runs, and shows the first lines it found. Under 50 words is an error, because that is what an AI crawler sees.

4. Don't blame the site for someone else's outage

The Recognized checks depend on outside services. On launch day, Common Crawl's index kept timing out on our requests. If the checker counted that as a fail, every site's score would have dropped for a reason that had nothing to do with the site.

So when a source does not answer, its check is left out of the total and the page says "did not answer, not scored." A score should only move when something about your site changes.

5. Free to run, on purpose

The obvious version of this tool asks ChatGPT and Gemini what they say about your brand. That is useful, but every check would cost money, and a free tool that costs money per use either gets switched off or gets a sign-up form.

So the first version calls no AI model at all. It uses a small Cloudflare Worker on the free plan, Wikidata and Common Crawl (both free public services), keeps each result for an hour so repeat checks cost nothing, and limits checks per visitor. Asking Gemini directly is the next step, on Google's free tier with no billing account attached, so it can never charge anything. When the daily limit runs out, that part will say "back tomorrow" and the rest will keep working.

6. A server that fetches any address needs guardrails

A tool that fetches any address a visitor types can be pointed at places it should never reach, like a company's internal network. The checker only fetches public addresses. It checks each address as written, then looks up where the name points before connecting, and refuses private and local addresses, including tricks like a public name that points back to the server itself. It follows 10 redirects at most, waits 8 seconds at most, and reads a limited amount of each page.

Its server also logs nothing. The address you check is fetched, scored and kept for an hour, and the deploy fails if logging is ever turned on for the Worker.

7. Reuse what already works

The site already had a Social Preview Debugger that fetches pages safely and reads their tags, robots.txt and structured data. The checker reuses that code rather than copying it, so a fix in one tool fixes both.

How it was tested and shipped

Each scoring rule has a test, and I broke each rule on purpose to make sure its test fails. One test passed when it should have failed: no test covered a crawler that robots.txt blocks and the server also refuses, which the score should count only once. I added that case before shipping.

It went live in two steps: first the Worker on its own, with no page pointing to it, then the page. The first Worker upload failed because the bundled file had the wrong name. The old version stayed live, nothing broke for visitors, and the fix was one line.

The live site check now runs a real check of robertoiriondo.com through the tool every time it runs, and fails if no score comes back. The page scores 99 on mobile and 100 on desktop in Lighthouse, with 100 for accessibility, best practices and SEO.

What it found on my own site

With the host back to normal, robertoiriondo.com scores 89, "Ready for AI search." Every crawler gets in, the homepage has about 2,900 words readable without JavaScript, and the structured data names me, my company and my profiles.

The biggest gap is outside the site: no Wikidata item lists robertoiriondo.com as its official website. Google and AI models use Wikidata to tell people and companies with similar names apart. That is now on my list.

What I'd measure next

• Where scores land. Each check sends the score and its band (0 to 39, 40 to 64, 65 to 84, 85 to 100) to analytics. If most sites land low on Reach, the next feature should explain firewall settings better.
• Which fix comes first most often. That is the most common real problem, and the best topic for the next guide.
• Whether people check again. A second check after a fix is the clearest sign the tool helped.

How to check your site

1. Open the AI Visibility Checker and enter your domain or any page.
2. Start with "Fix these first." The fixes are ordered by how many points each one costs you.
3. Look at the crawler table. Any "Refused" or "Challenged" next to an "Allowed" in robots.txt is your firewall or bot protection.
4. Fix, then press "Check again now" to see the new score.

If the checker cannot reach a site, the page lists how to check the same things by hand.

FAQ

Why did my site disappear from ChatGPT or Google AI answers? Usually because the AI tool could not reach or read the page when it looked. The common causes are a server or firewall that turns crawlers away, robots.txt rules that block the search crawlers behind AI answers, a noindex tag, or a page that only shows its text after JavaScript runs. Outages on your host can do it too, even when your code is fine.

Does blocking GPTBot remove my site from ChatGPT answers? No. GPTBot collects pages for model training. ChatGPT search uses OAI-SearchBot, and ChatGPT reading a page for a user uses ChatGPT-User. You can block GPTBot and still be cited, as long as those two can reach the page.

Is the AI Visibility Checker free? Yes. It needs no sign-up and calls no AI model, so it costs nothing to run. It uses a small Cloudflare Worker on the free plan plus Wikidata and Common Crawl, which are free public services.

What score should my site get? 85 or more means search engines and AI assistants can reach, read and recognize the site. Most of the points are about reach, so a low score usually means a crawler is being turned away, which is also the quickest thing to fix.

Roberto Iriondo is a marketing leader for AI companies, full-time or fractional. He co-founded Towards AI, acquired in 2022, and builds AI agents and tools at Generative AI Lab, which keeps his technical skills sharp.

Let's connect on LinkedIn or see how I work with AI teams on marketing.

← More notes