Is your website ready for AI crawlers? 6 technical checks

Last week we scanned 50 websites. Different industries, different countries, mostly companies doing good work in their field. 45 could be evaluated, the rest turned the scanner away or did not answer at all.

All 45 had at least one gap.

This article covers the six points we ran into most often. For each one it says what it is, how to look it up yourself and what to do about it. No technical knowledge needed, a browser is enough.

1. Do AI crawlers have access at all

At the root of your website sits a file called robots.txt. It states which crawler may fetch which pages.

At one event company it held five entries: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, all blocked. The company had not done this on purpose. A setting in Cloudflare had added those lines automatically.

How to check: Type your address and add /robots.txt. In the text that appears, look at which user agents carry the line Disallow: /.

What to do: If you want to appear in AI answers, those entries have to go. If you would rather not serve as training material, they can stay, and then the visibility goes too. You cannot have both at once.

2. How much text does the server deliver

A page has two states. The one you see in your browser and the raw one the server sends. Most of the time they are the same. Not always.

Most AI crawlers do not run JavaScript. They take what the server sends and move on.

Of the 45 sites, 7 had fewer than 300 words on the homepage. One had none at all: the server sent the same empty shell for every address, and the content only appeared in the browser. To AI systems that website consisted of a title.

How to check: Right click on your page and choose "View page source". In the window that opens, press Ctrl+F and search for a sentence that appears on your page. If you cannot find it, that text is not there for machines. "Inspect" is the wrong tool here, it shows the state after the browser has done its work.

What to do: If the text itself is missing, a few paragraphs and proper headings are often enough. If the text is on the page but does not arrive in the raw state, it comes down to the settings of the system you are using.

3. Are your company details machine readable

Structured data is extra information in the page source that machines read directly. Company name, address, phone, opening hours, services.

On 9 of the 45 sites structured data was present but the company details inside it were incomplete. On many there was none at all.

One distinction gets missed often: having the address visible on the page is one thing, marking it up so machines can read it is another. Without the second, systems collect the details from other sources.

How to check: Search the page source for application/ld+json. No match means there is no structured data.

What to do: At minimum Organization, or LocalBusiness for a company with a service area, each with address and phone. Product or service pages deserve their own.

4. Does every page have its own title and description

The title appears in the browser tab and in the search result. The description is the two lines below it.

At one manufacturer, 45 of 47 pages carried the same description, and it was a promotional text left over from the template the site was built with. The company makes compressors, the description talked about vape products.

Leftovers like this are more common than people expect. On sites built from ready-made templates, the sample text often stays in place.

How to check: In Google, type site: followed by your address, for example site:example.com. Read the descriptions under the results. If the same text repeats or does not fit, that is where the problem sits.

What to do: Give every page its own title and its own description. Put the subject of the page in the title, and the location where that helps.

5. Is the heading structure sound

Each page should have exactly one H1, the main heading. Below it H2, below that H3.

On 10 of the 45 sites the homepage had no H1 or had several. One site came with six H1 headings, all of them titles of product cards.

How to check: Search the page source for <h1 and count how often it appears.

What to do: One H1 that says what you do and where you are. Subheadings in order.

6. llms.txt

A text file that describes your website to AI systems in brief. It is a proposal and not a widely used standard.

In fairness this point sits last: next to the other five it carries little weight. Creating one takes a few minutes, and we have not seen a website become visible because of this file.

How to check: Add /llms.txt to your address.

What to do: A company description, your services and the addresses of your main pages. Keep it short.

What these six points do not solve

Technical fixes make a website readable. Being readable does not yet mean having something to say.

On some of the sites we checked the technical side was fine, but the page did not clearly state what the company does, where it works and for whom. Machines can read that page, they simply find nothing worth quoting.

There is also this: AI systems do not draw their answer from your website alone. Your Google Business Profile, directory entries and reviews count as well. That is good news, because all of it is in your hands. What matters is that the same details appear everywhere.

If you want to check your own site

Our Free SEO & GEO Check looks at most of these points for your homepage and states what it measured. It takes a few seconds and needs no signup.