Traffic from ChatGPT jumped! Is it because its crawlers now run JavaScript?

On September 25, 2026, early in the morning Japan time, OpenAI's crawlers changed how they read my sites. I saw it in the server logs. OAI-SearchBot (search) and GPTBot (training) stopped just fetching HTML and started running JavaScript and rendering pages.

The accepted wisdom has been that AI crawlers don't run JavaScript. The Vercel and MERJ study from December 2024 said none of OpenAI's three bots executed it. What I'm seeing is a change from that.

This post has the numbers from the logs first. After that comes a list of what I would do now if I wanted ChatGPT to mention a site. I couldn't find any announcement from OpenAI, so where I'm guessing, I say so.

What happened

The main site is @SOHO (atsoho.com), a Japanese freelance marketplace built on Next.js with the App Router. A few of my product sites run on the same server. I did not trust the user agent alone. Every request was checked against OpenAI's published IP ranges (gptbot.json, searchbot.json, chatgpt-user.json), and anything outside them was dropped as a fake.

Requests per day on atsoho.com

Date	OAI-SearchBot	GPTBot	ChatGPT-User
Sep 10	1,108	34	13,538
Sep 22	1,253	83	657
Sep 24	7,082	5	911
Sep 25	41,444	28,484	1,645
Sep 28	14,339	152,009	1,071

OAI-SearchBot jumped at 4 a.m. JST on September 25, and GPTBot took off at noon the same day. Until September 24 GPTBot came from one IP a day. From September 25 it came from 36 to 131.

Most of the new traffic was prefetch

When I looked inside, almost all of the extra requests were Next.js link prefetches, the ones with ?_rsc= plus five random characters at the end. That string is not in the HTML. It only shows up when the page is rendered and its JavaScript runs.

・September 22: 80% of OAI-SearchBot requests were plain HTML fetches, 2% were prefetches ・September 28: 84% of OAI-SearchBot requests and 97% of GPTBot requests were prefetches

So 150,000 requests a day does not mean 150,000 pages read. Most of it is what a rendered page fires off on its own.

How many pages actually got rendered

Each prefetch carries the page that triggered it in the Referer. Counting distinct Referers tells you how many pages the bot rendered.

Date	Rendered by OAI-SearchBot	of which job pages	Rendered by GPTBot
Sep 18	56	2	17
Sep 25	861	108	1,994
Sep 28	539	36	4,395

About 10x for search, and well over 100x for training.

They're rendering HTML they already have

Of the 4,309 pages GPTBot rendered on September 28, only 28% had been fetched from my server by anyone that day. My HTML is not cached at the CDN, so the CDN did not serve it either. My read is that OpenAI re-renders HTML it stored earlier in its own headless browser, and the only thing that reaches my server is the prefetches that fire during that render.

OpenAI's own docs say that if a site allows both bots, it may use the results from one crawl for both purposes to avoid duplicate crawling. GPTBot rendering what OAI-SearchBot fetched fits that.

Is this happening everywhere?

On the same server, all four Next.js sites that have articles saw OAI-SearchBot go up 5 to 15x on September 24 and 25. They run different Next.js versions and deploy differently, and I did not change anything on my side that week. The change came from OpenAI.

But on Cloudflare Radar, OpenAI's share of all AI bot traffic stayed flat through September. It looks like rendering is only switched on for some sites.

It reaches static sites too. On one of my sites that doesn't use Next.js, OAI-SearchBot pulled images and CSS on September 25, which is what rendering looks like. A static page does not fire prefetches, though, so the request count barely moves. The spike on the Next.js sites is mostly prefetch amplification.

What else changed at the same time

Around the same time, visits from ChatGPT to @SOHO's job pages went up, and daily signups rose about 1.7x. The number of people coming from ChatGPT did not change. Where they landed did. It used to be mostly blog posts, and from about September 25 it was mostly job pages. That lines up with job pages rendered by OAI-SearchBot going from 2 a day to 36 to 108. I cannot prove one caused the other.

What to do if you want ChatGPT to mention you

These six come from the logs above and from OpenAI's crawler docs.

1. Let OAI-SearchBot in

OpenAI's docs say sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. They recommend allowing OAI-SearchBot in robots.txt and allowing requests from their published IP ranges.

Three things to check:

・robots.txt: make sure OAI-SearchBot isn't disallowed ・WAF and bot protection: Cloudflare bot rules or your own rate limits might be blocking OpenAI's IPs. My own server was returning 403 to some of their requests on September 22 to 24 ・IP lists: OpenAI updates them. This time gptbot.json changed on September 22 and chatgpt-user.json on September 25. If you keep an allowlist by hand, pull it from the list URLs instead

If you do not want your content used for training, you can block GPTBot alone and keep search.

2. Put the content in the first HTML

OpenAI has started rendering, but only for some sites, it seems. Everywhere else, only the initial HTML gets read. And even where rendering happens, it starts from stored HTML.

Anything you want quoted, like the body, title, prices and dates, is safest in HTML that's readable without running JavaScript. In Next.js that means server rendering or static generation. On any site, avoid loading the main content from an API after the page opens.

3. Link to the pages you want read

The bots were reading prefetches inside rendered pages. The more internal links point at a page, the more chances it has to be read.

Google works the same way. One of my other product sites has 520 articles, and only 2.5% were indexed. The article list was paginated and Google never went past page 1, while 632 product pages linked to each other 37 times each and to the articles zero times. About 90% of Google's crawls went to those product pages.

・Do not bury articles deep in pagination ・Link related pages to each other, like a product page and the articles about that product ・Do not stop at the sitemap. Pages that only exist in the sitemap get pushed back

4. Don't block Next.js prefetches

Next.js has a check that turns prefetching off for known bots. The list has Google, Bing and others, but not OpenAI's bots (I checked the source in Next.js 14.2 and 16). So when an OpenAI bot renders a page, the prefetches go out as usual, lots of them.

I wouldn't block ?_rsc= in robots.txt or answer it with 429. A prefetch response contains the linked page's content, so for a rendering crawler it's one of the ways it reads your pages. Get the rule slightly wrong and you can block the pages themselves.

If server load becomes a problem, turn prefetch off only for the navigation links that appear on every page (prefetch={false}). That cuts the traffic and keeps the content and normal links intact. On my site most prefetches came from the category links in the global nav.

5. Tell machines how fresh things are

On @SOHO, ChatGPT sent people to job posts that had already closed. I cannot tell from outside how old ChatGPT thinks the information is. To avoid sending people to stale pages, put the published date, updated date and whether it's still open in both the structured data (JSON-LD) and the page itself. For job posts, one way is to mark only open posts as JobPosting and give closed ones a datePublished.

6. llms.txt can wait

I put llms.txt on @SOHO on September 20. OpenAI's bots read it once every few days. So far it does not look like it decides whether ChatGPT mentions you. It does not hurt, but I wouldn't put it ahead of 1 to 5.

How to check this on your own site

If you have server logs (nginx or similar), you can check all of this yourself.

・Identify the bots: match on the user agent (OAI-SearchBot, GPTBot, ChatGPT-User) and check the IP against OpenAI's three lists. An IP that is not listed is a fake ・Is it rendering: look for _rsc= in the URL (Next.js). On a static site, see whether the bot is fetching images and CSS ・Pages rendered: count distinct Referers on the _rsc= requests. Don't judge by request counts ・Traffic from ChatGPT: ChatGPT adds utm_source=chatgpt.com to its links. The Referer misses some of it, so count on the request URL ・Compare with everyone else: pull daily shares per AI bot from the Cloudflare Radar API

If a site is served only from Cloudflare, there are no server logs. Cloudflare's GraphQL Analytics API (httpRequestsAdaptiveGroups filtered by user agent) gives you counts by day and path, even on the free plan.

What I still do not know

・How OpenAI picks which sites to render ・When and how it fetches the HTML it renders ・How much rendering actually affects whether ChatGPT mentions a site

I'll add to this post as I find out.

References

・OpenAI, Overview of OpenAI Crawlers

https://developers.openai.com/api/docs/bots

・Vercel, The rise of the AI crawler (December 2024)

https://vercel.com/blog/the-rise-of-the-ai-crawler

・Next.js, Prefetching

https://nextjs.org/docs/app/guides/prefetching