Agentic AI

Your $300K Website Scores 100 on Lighthouse. AI Search Still Can’t Read It.

Why expensive startup websites miss the basics that search engines and AI answer engines rely on, and how a weekly AI audit catches them.

By Roberto Iriondo · October 17, 2026 · First published on Towards AI

An AI startup spends $100,000 to $350,000 or more on a new website. An agency designs and builds it. The pages load, the forms work, and the launch post goes out. The team moves on, and nobody looks at the files behind the site again.

Six months later, buyers ask ChatGPT about the category. The startup isn’t in the answer, or the answer describes a product it no longer sells. If someone runs a speed test, the site still scores in the 90s.

I’ve seen versions of this in my work with AI companies. Then I found it on my own site. It scored 97 to 100 on Lighthouse, and a weekly AI audit still found a sitemap from 2018, a robots.txt file pointing search engines to a university’s sitemap, and a contact email hidden from crawlers. A site that works for visitors can still be hard for search engines and AI to read.

This article covers what the audit found, why it happens even on expensive sites, how it shows up on Webflow, Framer, WordPress and custom builds, and how to check your own site.

Key takeaways

• A high Lighthouse or PageSpeed score measures speed and technical basics. It does not check whether search engines and AI can read your site correctly.
• The files that matter most are plain and easy to forget: sitemap.xml, robots.txt, structured data, and the facts your site states about your company.
• Every platform handles some of this automatically, and each one leaves a different gap.
• AI answer engines repeat what your site says about you. Outdated facts on your site become outdated answers.
• A weekly AI audit that reports problems, while a person decides the fixes, catches these issues early and costs very little.

Why do expensive startup websites still have these problems?

Because nobody owns the site after launch. The budget pays for design, build and handover. The boring files are created once and then left alone.

The scope ends at launch. Agencies are paid to deliver a site that works and looks right. The sitemap and robots.txt come from a template, a plugin or a copy of another project. Once the site is live, the contract is done.

The site changes, the files don’t. In the first year, a startup adds pages, renames its product, changes its positioning, and hires new people. The structured data and meta descriptions still describe launch day.

The checks people use measure the wrong things. Most founders judge a site by whether it works. When someone does run a test, it is usually a speed test like Lighthouse or PageSpeed Insights. Those scores can’t tell you that your sitemap lists pages that no longer exist, or that ChatGPT is quoting last year’s tagline.

Everyone owns a piece, so nobody owns the whole. Marketing owns the copy. Engineering owns the product. The agency owns nothing after handover. The files that connect the site to search engines and AI sit in the gap between them.

What did the AI audit find on my own site?

My site is a small consulting site. It is fast, clean, and built by someone who advises on marketing/SEO for a living. The audit read the site’s code, ran Lighthouse, and listed 14 findings ranked by impact. Six of them apply to almost any startup site.

1. The sitemap was last updated in 2018

A sitemap (sitemap.xml) is a file that lists the pages you want search engines to find.

Mine had dates from 2018. It listed a domain with a typo in it, a duplicate of the homepage, links to page sections that search engines ignore, and a file that no longer existed. The pages that mattered now were missing.

Why it matters: search engines use the sitemap to decide what to crawl. A wrong sitemap sends them to dead pages and hides the ones you want found.

Check your site: open yourdomain.com/sitemap.xml (most sitemap URLs fall here, but it depends on how it was built). Are the dates recent? Is every important page there? Is anything listed that shouldn’t be, like test pages, old campaigns or staging URLs?

2. The robots.txt file pointed to another site

A robots.txt file tells crawlers which parts of your site they may visit and where your sitemap is.

Mine had been copied from a university department’s site years ago. It pointed search engines to that department’s sitemap, blocked folders that didn’t exist on my site, and blocked a few crawlers for no clear reason. It didn’t block AI crawlers, which was good, but that was luck, not a decision.

Why it matters: robots.txt is one of the first files a crawler reads. If it points elsewhere or blocks the wrong paths, everything after it suffers.

Check your site: open yourdomain.com/robots.txt. Does the sitemap line point to your own domain? Does it block anything important? Look for “Disallow: /” on its own, which blocks the whole site and is often left over from a staging setup.

3. The contact email was hidden from crawlers

The email links on the page were Cloudflare email-protection links saved into the HTML. Visitors could still click them, but crawlers and AI engines saw a scrambled link instead of an email address. The links would also break if that Cloudflare feature were ever turned off.

Why it matters: if a machine can’t read how to contact you, it can’t tell anyone else. Some teams hide their email on purpose to reduce spam, which is a fair choice, but it should be a choice.

Check your site: view the page source (not the browser inspector) and search for your email address or contact page link. If you can’t find it in plain text, neither can a crawler.

4. The writing built other sites, not mine

I publish one article a week, and all of it lived on outside publications. My own site linked out to those articles but had no Article structured data and no article pages of its own.

Structured data is code (usually schema.org markup in JSON-LD) that labels what a page is about, such as a person, a company, a product or an article, so machines don’t have to guess.

Why it matters: content on other platforms builds their domain, not yours. Startups do the same thing with blogs on Medium or Substack, announcements only on LinkedIn, and docs on a separate platform.

Check your site: where does your best content live? If it lives elsewhere, publish it on your own domain first and add a canonical link on the copies that points back to the original. This article follows that rule.

5. Old facts in the structured data

The structured data on my site listed job details that no longer matched my current work. Visitors rarely see this code. AI answer engines do.

Why it matters: AI answer engines repeat the facts your site gives them. For a startup, this shows up as an old tagline, old product names, past pricing, a founder who left, or a funding stage from two rounds ago, sitting in the structured data, the About page, the footer or the press kit.

Check your site: ask your developer for the structured data on your homepage and About page, or view the page source and search for “application/ld+json”. Read it, along with your meta description, as if you were an AI deciding how to describe your company.

6. An FAQ written for search engines, not people

The FAQ answers repeated the same key phrases two to four times each. It read as if it had been written for a ranking algorithm.

Why it matters: repeated phrases make answers harder to read and harder to quote. AI answer engines tend to use short, direct answers that state one fact clearly.

Check your site: read your FAQ out loud. If an answer repeats your main keyword more than once, rewrite it in two or three plain sentences.

Does this happen on Webflow, Framer, WordPress and custom builds too?

Yes. Every platform creates some of these files automatically, which helps, and is also why teams stop looking at them. Each platform’s defaults leave a different gap.

Platform features change often. The notes below describe how each platform works at the time of writing. Check the platform’s current documentation before you change anything.

Webflow

Webflow is one of the most common choices for funded startups because designers can build and edit without engineers.

• The sitemap includes more than you think. Webflow generates the sitemap automatically and includes published static pages and CMS items unless you exclude them. Old campaign pages, thank-you pages, test CMS items, and utility pages end up listed.
• The staging domain can be indexed. Every Webflow site has a yoursite.webflow.io address. If indexing of that subdomain isn’t turned off, search engines can find a second copy of your site.
• robots.txt is a text box. It sits in the SEO settings, and it is often empty, copied from another project, or left with rules from before launch.
• Structured data is manual. Webflow doesn’t add schema for you. It goes in through custom code, and on CMS templates the fields have to be mapped, or every post ends up with the same markup.

Check: which pages are excluded from the sitemap, whether the webflow.io subdomain is set not to be indexed, and what structured data your CMS templates output.

Framer

Framer is popular with early-stage teams because a site can go from design to live in days.

• Speed makes it easy to skip the settings. The site looks finished long before each page has its own title, description and social image. Pages can end up sharing the same title and description.
• A second domain exists. Framer sites also have a Framer-hosted address. Make sure only your main domain is meant to be indexed.
• Structured data is manual. Like Webflow, schema goes in through custom code.
• CMS pages use templates. If the template’s SEO fields aren’t set up, every article or case study gets generic metadata.

Check: the title, description and social image on each important page, not just the homepage, and what structured data each page outputs.

WordPress

WordPress still runs a large share of company blogs and marketing sites, often with many years of plugins and themes layered on top.

• Two sitemaps at once. WordPress creates its own sitemap, and SEO plugins like Yoast or Rank Math create another. Sites end up with both, or with the wrong one submitted to Google Search Console.
• The staging setting left on. WordPress has a setting that asks search engines not to index the site. It is often turned on during the build and forgotten at launch.
• Plugins and themes add pages. Tag archives, author pages, attachment pages and theme template pages can end up in the sitemap. On another WordPress site I work on, theme template pages were listed in the sitemap next to real articles.
• Caching that isn’t working. A caching plugin can be installed and still not serve cached pages, which slows the site for both people and crawlers.

Check: that there is one sitemap, that robots.txt points to it, that the “discourage search engines” setting is off, and which page types are listed in the sitemap.

Custom builds (Next.js, Astro and similar)

Startups with strong engineering teams often build the marketing site in the same framework as the product.

• Nothing exists unless someone adds it. Frameworks offer ways to generate a sitemap and robots.txt from code, but a developer has to set them up. Many sites have a hand-written sitemap from launch that was never updated. That is exactly what happened on my site.
• Content that needs JavaScript may be invisible. If the main text only appears after JavaScript runs in the browser, some crawlers won’t see it. Google renders JavaScript, but many AI crawlers read only the raw HTML.
• Preview deployments get indexed. Hosting platforms create preview URLs for each change. If they are public and not blocked, search engines can find them.

Check: view the page source and confirm the main text is in it, that the sitemap is generated from your real pages, and that preview URLs are not indexed.

HubSpot CMS, Wix and Squarespace

• HubSpot CMS is often used for the blog and landing pages next to a site on another platform. That means more than one sitemap and more than one set of SEO settings to keep consistent.
• Wix and Squarespace handle the basic files well by default. The gaps are usually page-level titles and descriptions and missing structured data.

Docs, blogs and other subdomains

AI startups rarely have just one site. The docs may run on Mintlify, GitBook or ReadMe, the blog on Ghost or Substack, the app on its own subdomain, and a status page somewhere else.

Each subdomain has its own robots.txt and its own sitemap. For technical products, the docs are often the pages AI engines read and cite most, so a docs subdomain that blocks crawlers or lists outdated pages affects how AI tools describe your product. Some docs platforms now also generate an llms.txt file automatically (more on that below).

Check: list every subdomain you run, then open robots.txt and sitemap.xml on each one.

What every platform has in common

• The platform handles the format of the files. The content and the decisions are still yours.
• Staging and preview copies need to stay out of search.
• After a rebrand or redesign, old URLs need redirects, or links and rankings are lost.
• Someone has to look at these files after launch. No platform does that for you.

Why does this matter for GEO?

Generative engine optimization (GEO) is the work of making your site easy for AI answer engines, such as ChatGPT, Perplexity, Gemini, Claude and Google’s AI Overviews, to find, read and describe correctly.

The same files that guide search engines guide AI tools:

• If a crawler can’t reach a page, the page can’t shape the answer. AI answers draw on pages their crawlers can read and on search indexes. A blocked path or a missing sitemap entry removes you from the material they work with.
• AI tools repeat what your site says. If your structured data, About page and meta description describe last year’s company, the answer will too.
• Consistency helps. When your site, your structured data, your LinkedIn page, your docs and your profiles on sites like Crunchbase state the same facts, AI tools have fewer conflicting sources to choose from.
• Decide on AI crawlers on purpose. Your robots.txt can allow or block individual AI crawlers. Blocking them may keep your content out of AI answers. Crawler names and rules change, so check each provider’s current documentation instead of copying an old list.
• llms.txt is optional. It is a proposed plain-text file that summarizes your site for language models. It is not a standard, and support varies between AI tools. It is cheap to add, but it does not replace a correct sitemap, robots.txt and structured data.

The tools will keep changing. The basic rule will not: machines can only describe you correctly if your site states the facts clearly and they can read it.

What got fixed, and how long did it take?

Most of the findings were rated small effort, and the main ones were fixed in a single working session:

• The sitemap was rewritten to list only the pages that should be indexed.
• The resume page and a duplicate privacy page were set to noindex.
• robots.txt was replaced. It now allows all crawlers and points to the site’s own sitemap.
• The featured writing list now uses Article structured data.
• The main contact link was changed to a plain email link.

Two items are still open: a few remaining contact links, and article pages on the site itself. This article is the first step on the second one.

The fixes were not the hard part. Knowing the problems existed was.

How can you run your own weekly AI audit?

You don’t need a large tool. You need an AI assistant that can read your site’s files, a written checklist, and a habit.

1. Give the AI access to what it needs. For a custom build, that is the code repository. For Webflow, Framer or WordPress, it can read the live sitemap, robots.txt and page source, plus any exports the platform allows.
2. Write the checklist down. Sitemap, robots.txt, structured data, titles and descriptions, contact details, facts that should match your current positioning, and speed. Keep it in a file the AI reads every week, so each run checks the same things.
3. Ask for a ranked report. Each finding should say what is wrong, where, why it matters, the suggested fix and the effort.
4. Compare with last week. The value builds up over time: what got fixed, what is still open, and what is new.
5. Let it report, not change. A person, or your developer, decides what gets fixed and when. This keeps the live site safe.
6. Ask what it could not check. A report that lists its limits is easier to trust.

Pair the weekly report with Google Search Console, which shows what Google has actually indexed, and with a monthly test in ChatGPT or Perplexity: ask what your company does and compare the answer with your current positioning.

What did the audit say it could not check?

It listed its limits plainly: the live server’s response headers, the Cloudflare settings, Search Console data, one page that is only built at deploy time, and external links. It also noted that some Lighthouse findings came from a local test server and might not match the live host.

That list mattered. It showed where to look next, and it made the rest of the report easier to trust.

A quick checklist for founders

You can do this without a developer:

1. Open yourdomain.com/sitemap.xml. Check the dates and the pages listed.
2. Open yourdomain.com/robots.txt. Check that the sitemap line points to your domain and that nothing important is blocked.
3. View the source of your homepage. Check that the main text and your contact details are in it.
4. Search Google for site:yourdomain.com. Then search for your staging domain the same way (for example site:yoursite.webflow.io). The second search should return nothing.
5. Ask ChatGPT and Perplexity: “What does [your company] do?” Compare the answers with how you describe your company today.
6. List your subdomains (docs, blog, app, status) and repeat steps 1 and 2 on each.

Frequently asked questions

What is an AI website audit?

An AI website audit uses an AI assistant to review a site’s code and files against a checklist, such as the sitemap, robots.txt, structured data, page metadata and speed, and to produce a ranked report of problems and fixes.

Isn’t a high Lighthouse score enough?

No. Lighthouse measures speed and technical basics. It does not check whether your sitemap is current, whether robots.txt points to the right place, or whether your structured data states correct facts.

How often should a startup audit its website?

A light check every week, and a deeper review after a redesign, a rebrand, a platform change or a major launch.

Can AI fix SEO issues on its own?

It can find issues and draft fixes. A person should approve changes to a live site, because some fixes involve decisions, like which pages to hide or which crawlers to allow.

Which files matter most for GEO?

robots.txt, sitemap.xml, structured data, page titles and descriptions, and main content that is readable without JavaScript.

Do Webflow and Framer handle SEO automatically?

They handle the format of some files, such as the sitemap. They don’t decide which pages should be excluded, what your structured data should say, or whether your facts are current.

The site cost six figures. Keeping it readable takes a short report every Monday.

Most startup websites don’t need a rebuild. They need someone to look at the files behind them on a regular basis. Run the checklist above on your own site this week. If you want a second look, get in touch.

Roberto Iriondo is a fractional CMO and technical marketing advisor for AI companies, focused on website strategy, SEO, GEO and go-to-market. He founded Towards AI in 2019.

This article is reviewed as platforms change. Last reviewed October 2026.

Let’s connect on LinkedIn or see how I work with AI teams on marketing.

← More writing