We ran a measurement on our own marketing site that was uncomfortable enough to write up. Every public page, fetched the way a crawler fetches it, returned the same one-kilobyte shell: the same title, a description left over from a starter template, no headings, and about eight words of text.
In a browser those pages were fine — four hundred to a thousand words each, proper titles, real content. That gap is the whole story.
Two kinds of crawler, and only one of them renders
Googlebot executes JavaScript. It fetches the page, queues it for rendering, and comes back to the result. A single-page app is therefore indexable by Google, with caveats about rendering budget that matter more the newer your domain is.
The crawlers that feed language models do not render. GPTBot, ClaudeBot, PerplexityBot, CCBot and the rest fetch HTML and read what is in it. So do social unfurlers — the thing that builds the preview card when someone pastes your link into Slack or LinkedIn.
This matters more every quarter, because the question has changed. People used to search for "managed hosting Hyderabad" and scan ten links. Increasingly they ask an assistant, and the assistant answers from pages it could read. If your product only exists after JavaScript runs, it is not in that answer.
What we found when we looked properly
Measuring is worth doing before deciding anything. Fetch a page without a browser and read what comes back. Ours said:
- Identical titles on every URL. Not similar — identical, because the title was set by an effect that had not run.
- A description about component architecture, inherited from the template the app was scaffolded from.
- No headings and no body text. A div with an id, and a script tag.
- No Open Graph tags, so every link anyone shared unfurled blank.
- robots.txt returning HTML. There was no such file, so the SPA fallback answered with a page — and a 200 at that.
A dozen URLs with byte-identical HTML is also, incidentally, what duplicate-content handling is built to catch.
The fix is pre-rendering, and it is smaller than it sounds
You do not need to rewrite anything as a server-rendered framework. After the normal build, serve the output locally, visit each public route with headless Chromium, let React and its effects finish, and write the resulting HTML to a file per route. The web server then serves real HTML to anything that asks, and the app hydrates over it for people.
Three details make the difference between this working and half-working:
- Hydrate, do not re-render. If the entry calls createRoot on markup that is already there, it throws the pre-rendered DOM away and rebuilds it — a visible flash, and the markup the crawler read is replaced by markup it never saw. Call hydrateRoot when the root has children.
- Crawl your own links instead of listing routes. A hand-kept list of routes disagrees with reality within a month. Following links from the homepage also surfaces pages you forgot, and it found us eleven we had not listed.
- Check that your links are links. Ours were not. Card components navigated with onClick handlers, so nothing pointed at eleven pages of writing — not the crawler, not the pre-renderer, not a reader trying to open one in a new tab. Converting those to anchors took the crawl from fourteen routes to twenty-five.
What changed
Our homepage went from eight crawlable words to roughly fourteen hundred. Every page now carries its own title, description, canonical and Open Graph tags in the HTML, and the FAQ section is emitted as structured data built from the same array the page renders — so the two cannot drift apart.
The whole thing is one build script and about twenty lines of changes in the app. It took an afternoon, most of it verification.
Then say who may read it
Pre-rendering gives AI crawlers something to read. robots.txt decides whether they may, and the honest position is to pick one deliberately rather than leave it to a missing file. We allow them: if a model is going to answer questions about managed infrastructure, we would rather it had read our pages than guessed.
If you take the other view, that is a legitimate choice too — but make it in a robots.txt that exists, rather than by accident, in a site nobody can read either way.