Problem, approach, outcome
01Problem
My clients are mostly local businesses, and in 2026 their question changed. They still asked about Google, but they also asked whether ChatGPT or Perplexity could find them, and what those assistants would say if someone asked. The SEO tools I had answered a different question: rankings and backlinks, behind a subscription. None of them told me whether GPTBot was blocked in robots.txt, whether a page’s text only existed after JavaScript ran, or whether a business’s name, address and phone number were readable at all. I was checking those things by hand, one site at a time.
02Approach
Measure what the crawlers actually see. Each check sends real requests with each crawler’s own user agent, in parallel, and reads the raw HTML they get back, then compares it with what a browser gets so cloaking and JavaScript-only content show up. The robots.txt rules, the structured data, the sitemap, HTTPS and the security headers are all read from the live site, not inferred.
The harder work was judgement. A tool that cries wolf gets ignored, so I reviewed every heuristic for false alarms. Blocking an AI training crawler is often a deliberate choice, so it’s reported as information, not an error. A concise page isn’t a JavaScript problem. A redirect counts as reachable, and a link that times out, or a social network refusing a robot, isn’t called broken. Many of those rules came from scanning real client sites and finding the tool confidently wrong; each one became a ticket and a fix.
It’s plain PHP with no framework, running on the same shared hosting as this site, with a strict Content Security Policy and nothing stored about the visitor beyond the report itself.
03Outcome
Site Checker is where my SEO and AEO work starts. I run it on client and prospect sites before proposing anything, and its report is the checklist we work through. I ran it on my own sites first: it found 321enterprise.com with no HTTP-to-HTTPS redirect, no security headers and alt text on only half its images. All three are fixed, and it now scores 100 for crawlability, technical health and content readiness. Visitors who run a check can leave their email from the report, and some have. And building it taught me more about how AI crawlers and site health work than any article did.
