Key takeaways
- Run a site analysis in dependency order: access, then rendering, then structure, then content. Fixing content on a page crawlers cannot reach wastes the work.
- The single check most audits skip is fetching your own page as raw HTML. It takes one command and catches the failure mode that costs the most.
- AI tools are strong at breadth — every page, every tag, every broken link. They are weak at deciding which of the 400 findings actually matters.
- Crawl budget problems and thin-page problems look identical in a tool report and need opposite fixes.
- Re-run the analysis after the fix, not just before it. Items marked resolved in a tracker are regularly still failing at the edge.
Most site audits produce a list of 400 issues sorted by a severity score the tool invented. The list is accurate and close to useless, because it has no model of what depends on what. A missing alt attribute and a firewall rule blocking a crawler both show up as findings. Only one of them makes the other 399 irrelevant.
Running a useful analysis in 2026 means imposing an order the tools do not. Four layers, each of which has to work before the next one means anything.
Layer one: crawler access
Before anything else, confirm the machines you care about are allowed to fetch the page and are actually doing it. Three places to look, in this order.
Robots directives. A stray Disallow or a noindex left over from a staging environment is still one of the most common causes of a page that will never rank. Google's robots.txt documentation covers the precedence rules, which are less intuitive than they look.
Edge and firewall rules. This is the one no SEO platform inspects, because it lives in your CDN and not your HTML. We were blocking AI crawlers on this domain for months through a Cloudflare rule written to stop content scrapers. It did that. It also refused the assistants we wanted quoting us. Nothing in any audit report mentioned it.
Server responses. Redirect chains, soft 404s and pages returning 200 with an error body all corrupt everything downstream. A tool will flag the chain; it will rarely flag a 200 that renders "Loading…".
Layer two: what the machine receives
Modern crawlers differ in whether they run JavaScript. Google's does. Several AI crawlers do not, or do it inconsistently. If your content or structured data is written after page load, those fetchers see an empty shell.
The check is free:
Fetch the page as raw HTML rather than in a browser, then count the structured-data blocks and look for your H1. If the count is zero and the browser shows three, your schema exists only for visitors who run JavaScript.
We shipped that exact bug on this site until 17 July 2026. Every platform we paid for reported the schema as present, because every platform rendered the page first. The fix was moving the tags and JSON-LD into the server response, and it changed nothing a rank tracker measured on the day it landed.
Layer three: machine-legible structure
Once the content arrives intact, the question becomes whether a machine can parse meaning out of it. This is where AI-era analysis diverges most from the 2015 checklist.
| Signal | What a good tool checks | What it usually misses |
|---|---|---|
| Headings | One H1, no skipped levels | Whether the headings describe anything, or are all keyword questions |
| Structured data | Valid syntax, required properties | Duplicate graphs — two FAQPage blocks on one page is worse than none |
| Internal links | Broken links, orphan pages | Whether anchors vary, or repeat exact-match sitewide |
| Entities | Rarely checked at all | Whether the page names things a machine can resolve and cite |
Duplicate schema is worth singling out because it is common and counterintuitive. Emitting the same graph twice — usually once from the theme and once from the content — gives a parser two competing answers. On the sites we maintain it turns up more often than missing schema does, and the usual shape is a plugin emitting one FAQPage for the whole site while the post body emits a second — fifteen questions against six, with nothing to say which set is the page. The Schema.org getting-started guide is the reference worth reading once properly rather than copying snippets.
Layer four: whether the content earns the position
Only now does content analysis mean anything. Tools are genuinely good here at the mechanical end — thin pages, duplicate bodies, missing sections against what ranks. They are unreliable at the judgment end.
The judgment question no tool answers well: should this page exist? A set of near-identical location pages will each score respectably in isolation and suppress the domain collectively. Every tool grades the page in front of it. None of them grade the set.
Performance belongs to field data
Lab scores are a diagnostic, not a measurement. A synthetic run happens once, on one connection, from one location; the threshold it is graded against is a distribution. The Largest Contentful Paint reference puts the target at 2.5 seconds or less at the 75th percentile of page loads, and names the reason the two readings diverge: LCP in the field carries unload time from the previous page, connection setup, redirects and other time-to-first-byte delays a lab run never pays.
Practical rule: treat lab tools as a way to find the cause once field data tells you there is a problem. Running them the other way round produces months of optimisation aimed at a score nobody experiences.
What to fix first
| Finding | Priority | Why |
|---|---|---|
| Crawler blocked at the edge | Immediate | Nothing else you fix will be seen |
| Content or schema client-side only | Immediate | Invisible to fetchers that skip JavaScript |
| Duplicate schema graphs | High | Actively worse than having none |
| Near-duplicate page clusters | High | Suppresses the whole domain, not one URL |
| Thin content on pages with impressions | Medium | Real demand already exists — cheapest lift available |
| Missing alt text, minor tag issues | Low | Genuine, but nothing depends on them |
The internal-link row is worth a sentence of its own, because it is the finding tools report most confidently and act on least. An audit of all 95 posts on this domain found 43 with no internal links at all and 62 sitting below a three-link floor. Every crawler we ran flagged the orphans. None proposed the fix, which was a render-time link map rather than 62 hand edits, because that is a decision about how the site is built and not a property of any page.
The pattern in that table is the whole method: order by what other things depend on, not by the tool's severity colour. We used the same sequence on the Charleston Shark Teeth rebuild, where access and rendering fixes preceded any content work at all.
When two tools disagree, the disagreement is the finding
Running a second crawler over the same domain reliably produces different numbers, and the instinct is to decide which tool is right. That is the wrong question. The two tools fetched differently, and the way they differ usually names the problem.
One renders JavaScript and one does not — so a page rich in the first report and empty in the second is a rendering problem, not a tooling inconsistency. One respects a crawl-delay directive and one ignores it, so a large gap in pages discovered points at throttling rather than at missing links. One follows redirects and one records them, so a chain invisible in the first report is sitting in the second.
The practical move is to keep both and read the delta deliberately. A single crawler gives you one view of the site and no way to know what that view excluded. Two give you a comparison, and the comparison is where the edge-level failures surface — precisely the failures that no single report will ever contain.
Re-check after the fix, not only before it
The step most audits skip is verification. An item marked resolved in a project tracker means somebody changed a file. It does not mean the change reached the machine doing the fetching.
On our own remediation passes, a meaningful share of items marked fixed were still failing when re-tested against raw HTML. A cached response was still being served at the edge. A build had shipped the change to one route and not the template it shared. A header directive set at the CDN was quietly overriding what the application returned. In each case the source code was correct and the fetched page was not.
Build the re-check into the work rather than treating it as optional. Fetch the affected URLs as raw HTML after deployment, confirm the specific thing you changed is present in the response, and purge the cache before concluding anything. This is also the point at which a fix that silently broke something adjacent becomes visible, while the change is still fresh enough to reason about.
If you want the access and rendering layers checked against your own domain, that is what our free AI visibility audit runs. The tool-selection question is covered separately in what AI SEO tools actually change, and the underlying service sits under AI search optimization.
Frequently asked questions
What should an AI SEO site analysis actually check?
Four layers in dependency order: crawler access including edge and firewall rules, what the raw HTML response contains, whether the structure is machine-legible, and finally content quality. Most tools start at layer four because that is where the findings are easiest to generate.
How often should I run a site analysis?
A full pass quarterly, plus a targeted re-check after every deployment that touches templates, routing or the CDN. The failures that cost the most are usually introduced by a change rather than accumulating slowly.
Why do two SEO tools give different results for the same site?
Because they fetch differently. One renders JavaScript, another does not; one respects a crawl-delay, another ignores it; they sample different page sets. When two tools disagree, the disagreement itself is the finding — check what each one actually requested.
Can AI tools find technical SEO problems automatically?
They find candidates well and rank them badly. Breadth is genuinely valuable — no human checks 5,000 pages for missing tags. The ordering, and anything living outside the HTML such as firewall rules or cache behaviour, still needs a person.
What is the difference between a crawl error and an indexation problem?
A crawl error means the fetcher could not retrieve the page. An indexation problem means it retrieved the page and chose not to keep it. The second is usually a quality or duplication signal, and adding more internal links will not fix it.
Does site speed still matter for AI search visibility?
Indirectly but genuinely. Slow responses reduce how much of a large site gets fetched, and fetchers on tight timeouts may abandon a page before content arrives. Field performance data matters more than a lab score here.
