JavaScript rendering and AI crawlers

Googlebot

Google processes JavaScript pages in three phases: crawling, rendering and indexing. Pages are queued for rendering, and “once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript”. A page may wait in this queue for a few seconds, but it can take longer.

Officially documented · Google Search Central

Google still recommends server-side rendering or pre-rendering. The reason: “it makes your website faster for users and crawlers, and not all bots can run JavaScript.”

Officially documented · Google Search Central

AI crawlers

Observed in 2024-12, more than 9 months ago. Answer engines change often; re-test before relying on this.

In tests published in December 2024, Vercel and MERJ found that none of the major AI crawlers they examined rendered JavaScript. They named OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider) and Perplexity (PerplexityBot). Gemini, which uses Googlebot’s infrastructure, did render JavaScript. So did AppleBot.

Observed in practice · observed 2024-12 · Vercel & MERJ

This is a single industry study, almost two years old at the last review. Crawler behaviour can change without announcement. None of the companies named document whether their crawlers render JavaScript. Treat the study as the best available observation, not a specification.

What follows for a site

• Put the main content, title, description, headings and links in the HTML the server returns.
• Don’t rely on JavaScript to inject structured data, canonical tags or noindex for crawlers other than Googlebot.
• Check what a crawler receives: fetch the page without JavaScript (curl -A "OAI-SearchBot" …) and compare it with what users see.

How this wiki handles it

Every page is pre-rendered to static HTML at build time. Search, filters and theme switching are enhancements on top. The Markdown twin of each entry (/path.md) gives agents the same content without any HTML or JavaScript.