Crawler simulator: see your site through any bot's eyes
Different bots see different versions of your website. Googlebot renders JavaScript (eventually), but ChatGPT's GPTBot does not. Facebook's crawler reads Open Graph tags from static HTML, while LinkedIn's bot has its own quirks.
This crawler simulator lets you pick a specific bot and see exactly what HTML it receives from your page. Choose from search engines, AI crawlers, and social media platforms to test each one individually.
Understanding these differences is the first step to ensuring your content reaches every platform that matters to your audience.
Pick at least one crawler above, or use "Select all" to test every bot at once.
Test a specific crawler
How to use this tool
- 1
Choose a crawler from the list below (or select one from the category tabs)
- 2
Enter your page URL in the input field
- 3
Click 'Simulate crawl' to fetch your page with that bot's user agent
- 4
Review the HTML response - check for your page title, main content, and meta tags
- 5
Test multiple crawlers to compare how each one sees your site
How it works
When you select a crawler and enter a URL, we send a server-side HTTP request using that bot's exact User-Agent string. For example, selecting Googlebot sends the request with 'Mozilla/5.0 (compatible; Googlebot/2.1)' - the same identifier Google's real crawler uses.
Your web server receives this request and responds based on the User-Agent. If you have dynamic rendering, cloaking rules, or bot-specific logic configured, the response will reflect that. This is exactly what happens during a real crawl - we are replicating the full request/response cycle.
The returned HTML is then analysed for SEO elements: title tag, meta description, Open Graph tags, heading structure, canonical URL, structured data, and overall text content. Each element is scored, and the tool flags missing or problematic items with severity ratings.
When you select multiple crawlers, each one makes an independent request in parallel. The results are displayed side by side in a comparison table so you can spot differences - for example, a page that returns complete HTML for Googlebot but an empty shell for GPTBot.
Why this matters for your site
Every web crawler identifies itself with a unique User-Agent string. This string tells web servers which bot is making the request, allowing sites to serve different content to different crawlers - a practice known as dynamic rendering.
Search engine bots like Googlebot and Bingbot have varying levels of JavaScript execution capability. AI crawlers including GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot are primarily text-extraction bots that read raw HTML without executing scripts. Social media crawlers like Facebook's facebookexternalhit and LinkedInBot focus specifically on Open Graph and Twitter Card meta tags to generate link previews.
Understanding which bots can and cannot render JavaScript is essential for SEO strategy. A site that works perfectly for Googlebot may be completely invisible to AI platforms and social networks, costing you traffic, referrals, and AI-powered recommendations.
Frequently asked questions
Why do different crawlers see different content on the same page?▾
Each crawler has different capabilities. Googlebot can execute JavaScript using a headless browser, but social media crawlers like Facebook and LinkedIn only read raw HTML. AI crawlers like GPTBot and ClaudeBot also skip JavaScript entirely. The content each bot sees depends on what your server returns in the initial HTML response before any client-side rendering.
What is a User-Agent string?▾
A User-Agent is an HTTP header that identifies the client making a request. Bots use distinctive User-Agent strings so servers can identify them. For example, Googlebot sends 'Mozilla/5.0 (compatible; Googlebot/2.1)'. This allows servers to serve bot-optimised content when needed, which is the basis of prerendering and dynamic rendering strategies.
Should I serve different content to different crawlers?▾
Serving entirely different content to bots is considered cloaking and violates search engine guidelines. However, serving a pre-rendered version of the same content is an accepted practice called dynamic rendering. Google officially recommends this approach for JavaScript-heavy sites. The key is that the rendered HTML must contain the same content a human would see.
How often do AI crawlers visit my site?▾
AI crawler frequency varies significantly. GPTBot (ChatGPT) has increased its crawl rate by over 300% year-on-year. ClaudeBot and PerplexityBot are growing rapidly as well. Unlike search engines that may visit daily, AI crawlers tend to crawl in bursts when updating their training data or search indexes.
Can I block specific crawlers from my site?▾
Yes. You can use your robots.txt file to block specific crawlers by their User-Agent name. For example, adding 'User-agent: GPTBot' followed by 'Disallow: /' will prevent ChatGPT from crawling your site. However, blocking AI crawlers means your content will not appear in their responses, which may reduce your visibility.
Related free tools
What bots see
Enter any URL and see the raw HTML that search engine bots, AI crawlers, and social platforms receive - before and after JavaScript rendering.
Try it freeOG image checker and Open Graph debugger
Test your Open Graph and Twitter Card meta tags. Preview exactly how your links appear when shared on Facebook, LinkedIn, Twitter/X, and WhatsApp.
Try it freeSEO audit
Run a free SEO audit on any website. Check meta tags, headings, structured data, alt text, canonical URLs, and crawler accessibility.
Try it freeMake every crawler see your content
PageGlass automatically serves rendered HTML to all bots. No per-crawler configuration needed - one simple change covers Googlebot, AI crawlers, and social platforms.