Most AI crawlers do not execute JavaScript. They fetch your HTML, read what is in it, and leave. If your content only appears after a script runs, those systems see a page with a navigation bar and nothing else. Googlebot renders JavaScript. Most of the crawlers feeding AI assistants do not, and that difference is invisible from a normal browser.
How to check what they actually see in thirty seconds
Open your site. Right click and choose View Page Source. Not Inspect, which shows the finished result after scripts have run. View Source shows the raw HTML your server sent, which is what a non rendering crawler receives.
Now use your browser’s find function and search for a distinctive sentence from the middle of your page.
If it is there, you are fine. If it is not, your content does not exist to any crawler that does not run JavaScript, and that includes a significant share of what feeds AI answers today.
Which sites have this problem
Sites built as single page applications in React, Vue or Angular without server side rendering. Sites where the content loads through an API after the page appears. Sites using a headless setup configured for client rendering. Some page builders that assemble sections with JavaScript. Anything with content behind a tab, accordion or “load more” that fetches on interaction.
Ordinary WordPress sites are usually fine, because WordPress renders HTML on the server by default. It is worth checking anyway, because a plugin or a builder can undo that for specific sections without anyone noticing.
Why this is not simply an SEO problem
Google has rendered JavaScript for years, so teams that checked their Google rankings concluded the issue was solved. It was, for Google. The crawlers that assistants rely on are a different and more varied set, many of them simpler, and several of them do exactly one fetch with no rendering step.
The result is a site that ranks acceptably in classic search and is functionally invisible to AI retrieval. That is a genuinely new failure mode and it did not exist five years ago.
What to do about it
Render on the server
The complete fix. For a React or Vue site that means server side rendering or static generation. It is real engineering work and it is worth it, because it also improves load time and classic crawlability at the same time.
Do not hide content behind interaction
Accordions and tabs are fine as long as the content is present in the HTML and merely hidden with CSS. It is not fine when the content is fetched only when someone clicks. The first is a display choice. The second removes the content from the document.
Check the specific things you most want cited
Your service descriptions, your pricing explanations, your answers to common questions. These are the passages most likely to be retrieved, so they are the ones worth confirming are in the raw HTML.
Publish an llms.txt
A plain text description of your site at the root. It is a proposed convention rather than an established standard, and whether it is worth your time yet is a fair question, but it costs almost nothing to add.
This site as the worked example
Every word on palmprojects.com is in the HTML the server sends. There is no JavaScript framework between a crawler and the content. The only script is a small file handling the menu, the scroll behavior and the table of contents, and with JavaScript disabled entirely the site still renders and every link still works.
That is not an accident and it is checkable in the same thirty seconds described above. The Under the Hood page publishes the rest of the numbers.
Questions
Will Google penalize a JavaScript site?
No. Google renders it. The cost is elsewhere: slower indexing, a rendering budget that can leave pages waiting, and invisibility to non rendering crawlers.
How do I know which crawlers have visited?
Server logs, if your host gives you access to them. You can identify AI crawler user agents and see whether they arrive and what they request. WP Engine and most managed hosts provide log access on request.
Is this worth fixing on an otherwise working site?
If your content is not in the raw HTML, yes, and it is more urgent than most items on a typical technical list, because the cost is total rather than incremental. Being ranked lower is a degree. Being absent is a category. This is where an AI search engagement usually starts.
