Googlebot simulator: test how Google really sees your site

This test sends one server-side GET request with the header 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' and records what comes back: the status code, the response headers and the HTML body. No scripts are executed. Google crawls most sites with its smartphone agent, but the 'Googlebot/2.1' token is the same in both versions and is what servers match on.

What the test deliberately leaves out is Google's Web Rendering Service. Google renders pages later in an up-to-date version of Chromium, so content injected by JavaScript can still be indexed after a delay. Testing the raw response shows what Google has to work with before that happens, and what it is left with if rendering is deferred, a script fails or an API call is too slow.

Two results need careful reading. A 403 or a challenge page usually means your CDN or firewall verifies Googlebot by IP address or reverse DNS and has rejected our request because it does not come from Google's network. The real Googlebot would not be blocked. And if the response differs from what a browser receives, your server is varying content by user-agent, which is acceptable for pre-rendering as long as the content is equivalent.

Googlebot is Google's primary web crawler responsible for discovering and indexing content for Google Search. Unlike most other crawlers, Googlebot can execute JavaScript using the Web Rendering Service (WRS) - a headless Chromium-based engine maintained by Google's search team.

However, JavaScript rendering happens on a delayed schedule. Google's indexing pipeline works in two waves: the first wave indexes the raw HTML immediately, and the second wave renders JavaScript hours or days later. During this gap, your content may be invisible in search results. For time-sensitive content like product launches, news articles, or event pages, this delay can mean missing critical indexing windows.

Googlebot also operates under a crawl budget - the number of pages Google will crawl on your site within a given timeframe. JavaScript-heavy pages consume more crawl budget because rendering is expensive. Sites with thousands of pages may find that Google deprioritises rendering for less important pages, leaving them permanently unindexed.

Pre-rendering your pages ensures Googlebot sees fully-formed HTML on the first crawl, eliminating the two-wave delay and reducing crawl budget consumption. This is especially important for React, Vue, Angular, and AI-built sites where the initial HTML is often just an empty shell.