Walking my site like Googlebot: what two months of reading the code missed

For two months I tried to get my site into Google search. At best it got about ten impressions a day. Every attempt started the same way: my agent read the sitemap and the code, and both looked fine.

Then I just opened the site and watched it. The keywords appeared after the page did. So I told my agent to stop reading the code and walk the site the way Googlebot does. It built a crawler on Crawlee and Playwright that follows only real <a href> links and reads every page twice: as the server sent it, and after JavaScript ran. It is now a public repo you can run on your own site.

It found what reading never had:

• The home page sent 174 of its 255 links to pages marked noindex, and only 3 to stories.
• /stories had 82 links from the server and 22 after JavaScript. Every story link was gone, because the script had turned links into buttons.
• None of the 66 keyword pages had a link from a story.
• Every story page emptied itself while loading, from about 5,500 characters to about 540, then rebuilt.

The part I had not seen written anywhere: comparing the first and last state of a page hides that emptying. My home page went from 278 links to 306, which looks healthy. Only sampling during the load showed it falling to 23 in between.

Search Console also killed my agent's theory. It suggested hiding keyword pages with few stories, but 2-story pages were indexed while a 27-story page was never crawled. What lines up is age: all 11 stories from April to July are indexed, against 7 of 18 from October.

The crawler paid for itself the same night. Re-running it on the local build caught my agent's own fix breaking all four author pages, before they went live.

What I don't know yet is whether any of this brings visitors. The fixes went live today. Google's renderer waits for pages to settle, so the emptying may never have hurt. The lost links were real flaws, but they are not proven to be why so few pages were indexed. Links from other sites probably matter a lot too. I will update this story with what Search Console shows.

Check if this applies to you

Applies if: you own the site, its pages are sent as HTML, and JavaScript then fills or rewrites them in the browser. On a site with no client-side rendering this check means nothing.

These steps only read. In a headless browser, block your analytics hosts and any request that isn't GET or HEAD, so the check changes no data. Page views will still show in your server logs.

1. Pick the home page, a listing page and one deep page.
2. For each, count <a href> links in the raw HTML from curl.
3. Load it in Playwright with an init script that records document.links.length, document.body.textContent.length and document.readyState on every DOM change, using a MutationObserver. Wait for the network to go quiet.
4. Compare the raw links, the lowest point during load, and the final state. Ignore samples where readyState is loading: the HTML is still arriving, so low counts there mean nothing.

Has the problem: links in the raw HTML are gone from the final page, or, once the HTML has arrived, text or links stand at less than half of what the server sent, or the home page mostly links to pages whose <meta name="robots"> says noindex.

Doesn't have it: the final page keeps the raw links, text and links stay above half of what the server sent, and the home page links mainly to indexable pages.

Reproduce this

Prerequisites: a site you own (only crawl your own, this uses Googlebot's user agent and a real browser), Node.js 18+, Python 3, and a local build of the site you can crawl before deploying. Search Console access for the site makes step 4 possible.

Your agent will need from you: the site URL, your analytics hostnames, and which pages are meant to be noindex.

What this does not prove: it is an imitation, not Googlebot. It cannot show Google's rendering queue, its crawl budget or how much it trusts your site. It clicks and scrolls nothing. Google doesn't publish its viewport or how long it waits, so a 412x732 viewport and up to 15 seconds of waiting are my assumptions. My site has about 350 URLs, which took a few minutes at 3 pages at a time.

Steps:

1. Clone crawl-like-googlebot, then run npm install, npx playwright install chromium and npm test. The test uses a small local site with known flaws and must pass before you point it at a real one. It is built on Crawlee, Playwright, robots-parser and cheerio, and its rules come from Google's docs on crawlable links, JavaScript SEO and Googlebot.
2. Check what it could disturb. It blocks common analytics hosts and every POST, PUT, PATCH and DELETE on any host, which is where it differs from Googlebot on purpose. A GET that counts something still runs, so exclude it with --skip, as I do for my raw story files. Add other analytics hosts with --analytics. Page views still reach your server logs, so filter them out of your stats.
3. Run node crawl.mjs --site https://your-site.example --out crawl-first, then python3 analyze.py crawl-first, and read crawl-first/report.md. Keep this first report to compare against.
4. Inspect each URL with the URL Inspection API and compare its status with the crawl before you believe any cause. Treat each cause as a guess until this data backs it.
5. Fix what the crawl shows. What I changed: titles and cards became real links again, and those links still go through the app's router. The server's HTML stays on screen while the app builds its own version out of sight, and the two swap once the data has arrived. Story keywords link to their keyword pages. Rare keywords link to their only story instead of a noindex page. Those noindex pages still exist, because stories and old links can point at them; I only stopped sending the home page there.
6. Crawl the local build before deploying, and click through it in a real browser, because the crawler never clicks. Every click must add exactly one history entry, and modifier-clicks must still open a new tab.
7. After deploying, crawl production again and resubmit the sitemap. If your deploy only announces URLs that changed since the last sitemap, a local build may have already overwritten that baseline, so check it really sent them. Mine did not.

Verify: on the production crawl, compare against your first crawl. No page loses more than half of its text during load, except any you can explain. Listing pages keep every link to their items after JavaScript. Indexable keyword or category pages each have a link from a content page. The home page has no links to noindex content pages. There are no JavaScript exceptions, and all indexable pages are in the sitemap or left out on purpose. Then, weeks later, compare Search Console's status for the same URLs. Only that step tells you whether it helped with Google, and it may not.