Back to Blog
Angular DevelopmentNovember 11, 20268 min read

Is Your JavaScript Site Invisible to AI Search? How to Check and Fix It

DR

Doug Ribeiro

8 min read

A client-rendered Angular app looks complete in a browser and close to empty to any bot that does not run JavaScript. Before moving my portfolio to static prerendering, I measured that baseline the way such a crawler would. A blog post of just over a thousand words came back as an empty app-root, with the same generic title every page shared. The article only existed after Angular booted in a browser.

After the migration, the same URL returns about 1,300 words in the initial HTML, with its own title. That count includes the header and footer, but the difference is the whole article. The content itself did not change.

What the Crawlers Actually Do

Google is the famous exception, and it fooled a generation of us. Its documentation describes three phases: crawl, render, index. Pages wait in a queue until a headless Chromium can execute the JavaScript. So a client-rendered app can rank on Google, and that taught us that crawlers run JavaScript. The same guide still says server-side or pre-rendering is worthwhile because not all bots can run JavaScript.

The measurement I keep seeing cited for the others is Vercel's, from December 2024. It found that OpenAI's crawlers, Anthropic's ClaudeBot and the crawlers from Meta, ByteDance and Perplexity did not render JavaScript. ChatGPT's and Claude's crawlers did download JavaScript files, but did not execute them. Gemini, on Google's infrastructure, and AppleBot did render. That data is almost two years old, and I did not find a newer study of the same quality. Crawlers change, so verify against your own site and your own logs.

The One-Minute Test

Pick a real content page and a short phrase from the middle of it, with no punctuation, because HTML entities cause false zeros. Then ask for the page the way a simple crawler would. The grep counts matching lines, so zero means the phrase is not in the HTML that was served. A hit can come from a JSON state blob instead of the markup, so open the output and look if the count surprises you.

bash 
# Is the content in the HTML the server sends?
curl -sL https://example.com/some-article | grep -c "a short phrase from the middle"

# Is your host or CDN treating a bot differently?
curl -sL -o /dev/null -w '%{http_code}\n' -A "GPTBot" https://example.com/some-article

# Does a page that does not exist return a real 404?
curl -sL -o /dev/null -w '%{http_code}\n' https://example.com/this-page-does-not-exist

On the client-rendered baseline, the first returned 0 and the third returned 200. That last one is a soft 404. Single-page hosting usually rewrites every unknown URL to index.html, so bots get a success status and an empty shell for every dead link. It is the failure people miss because the page looks right in a browser, where the app renders a nice not-found screen.

Know Which Bot Is Which Before You Touch robots.txt

Blocking AI bots is often one decision made about three different things. The vendors separate them, and the distinction decides what you lose.

  • Training: GPTBot and ClaudeBot. Blocking these says your content should not be used to train models.
  • Search index: OAI-SearchBot and Claude-SearchBot. Block OAI-SearchBot and OpenAI says your site will not appear in ChatGPT search answers, though it can still show as a navigational link. Anthropic warns that blocking Claude-SearchBot can reduce your visibility in its search results.
  • User-triggered fetch: ChatGPT-User and Claude-User. These fetch a page because a person asked. Anthropic says blocking Claude-User can reduce visibility for user-directed search. OpenAI says ChatGPT-User is not what determines whether content appears in search, and that robots.txt rules may not apply to it.

So you can refuse training and still be findable. Here is what that looks like. My own site allows everything, because a portfolio wants to be read, but this is the file I would start from if I did not want to feed training:

text 
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Go read your current robots.txt now. If someone added a blanket AI block a few years ago, check whether it also catches the search bots. Also check any bot-protection setting at your CDN, which can block crawlers that robots.txt allows. The second curl above can catch a block keyed on user agent, though not one based on IP.

I Skipped llms.txt, and the Evidence Backs That

llms.txt is a September 2024 proposal by Jeremy Howard for a Markdown file that points language models to your key content. It is not a ratified standard. It is cheap, so people add it hoping for AI citations. The evidence is not on their side.

  • Ahrefs looked at server logs across 137,000 domains, all its own analytics customers, so it treats the 28% that publish the file as an upper bound. 97% of those files received zero requests in May 2026.
  • SE Ranking studied about 300,000 domains and found no measurable link between having the file and being cited.
  • Google's guidance says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. John Mueller has also said LLM systems cannot trust the file as a way to tell websites apart.

One honest counterpoint. Ahrefs found GPTBot and Claude Code near the top of the AI agents fetching these files, though GPTBot was only about 4.5% of fetches, and Chrome added an llms.txt check to Lighthouse's experimental agentic audits. If you run a documentation site that coding agents read, the file can earn its few minutes of work. For a marketing site, a blog or a portfolio, I would spend those minutes on the HTML. My site does not have one.

What the Fix Took on a Static Host

You do not need a Node server for this. My site lives on static hosting, and prerendering costs build time plus one real constraint: changing content now means a rebuild and a deploy. Angular has this built in: add @angular/ssr, set outputMode to static in angular.json, and give every public route a Prerender mode. Routes with parameters have to declare their values at build time, which is the part that takes thought:

typescript 
// app.routes.server.ts (simplified)
export const serverRoutes: ServerRoute[] = [
  {
    path: 'blog/:slug',
    renderMode: RenderMode.Prerender,
    getPrerenderParams: async () => postSlugs().map(slug => ({ slug })),
  },
  { path: '**', renderMode: RenderMode.Client }, // the shell I publish as 404.html
];

On React, the same decision lands on a framework: Astro, Next.js, or React Router in framework mode. Next.js static export drops per-request rendering, so check before you pick it. The tool does not matter. Decide per route: content that anyone can read gets HTML at build time, content that changes per request gets server rendering, and anything behind a login can stay client-rendered, because no crawler gets there anyway.

The build produces about 300 pages across four languages. The render was the easy half. Here is the rest:

  • A page-specific title and a unique description and canonical on every page, in the initial HTML.
  • Hreflang between real translations only. A page that falls back to English gets a canonical to the English URL and no alternates.
  • A sitemap generated at build time from the post data, and a build that fails if the post counts disagree.
  • One URL per page. Pick trailing slash or no trailing slash and redirect the other. Apache adds its own directory redirect, so I turned that off with DirectorySlash Off, added a rewrite to each folder's index.html, and kept directory listings off.
  • A real 404 status. A 404.html through ErrorDocument, not a rewrite to index.html.
  • Hydration that does not fight the static HTML. In Angular, look at provideClientHydration and its transfer cache. Anything that depends on the finished page, like scrolling to an anchor, has to wait for ApplicationRef.whenStable. The typical symptom is a link that scrolls before the section has loaded.

What Prerendering Does Not Promise

Readable HTML gets you considered. That is all it gets you. Nothing here guarantees that an AI answer will cite you. What I can say is that the failure is binary: content a bot cannot read cannot be cited from that fetch. Fix that first, because it is the only part of the AI search discussion you fully control.

After deploying, read your access logs instead of guessing. A grep for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User and PerplexityBot shows who visits and which status they receive. User agents can be spoofed, so treat it as a rough signal.

Open your site with JavaScript disabled, or curl it. Whatever you see is what a large share of the bots you care about see.

References

SEOAngularPrerenderingAIWeb Performance

Related Posts

Angular Development10 min read

Angular Performance Optimization: What Actually Moves the Needle

I have audited a lot of Angular apps and the same gaps appear every time. This is what I fix first, why, and in what order.

Angular Development8 min read

Angular Lazy Loading: Faster Apps With Less Effort

Lazy loading is the single highest-impact performance change I make on every Angular app I work on, and with standalone components it takes minutes per feature route.

AI & Development7 min read

Verification Debt: AI Made Writing Code Cheap, Not Trusting It

A coverage gate can look strict and enforce nothing. AI raises the volume of code, so proving that each check can fail matters more than adding new ones.