80% of Googlebot’s requests went to Next.js prefetches. Here is the one-line fix.

By Mehmet Apaydın ·

Sizefoto is a passport photo tool with about 900 pages in ten languages: one page per country and document, plus guides. It is a Next.js App Router site, fully server-rendered, and technically clean by every checklist: canonical tags, hreflang, sitemap, one H1, structured data. Yet Search Console kept listing 119 URLs as "Discovered, currently not indexed", and most of them were the guides, the pages with the most original writing on the site.

Where the crawl budget went

The answer was in a report most people never open: Settings, Crawl stats, "By file type". Over 90 days Googlebot made 12.9K requests. HTML was 15% of them. "Other file type" was 80%, 10.3K requests. The examples under it all looked the same:

https://sizefoto.com/en/guides?_rsc=u0Jl2VMmNatQE7DV
https://sizefoto.com/en/gb/visa?_rsc=Dz8GcTgvVhaIqvIA
https://sizefoto.com/ko/in/passport?_rsc=OzagwAilFo8pC1ee

Those are React Server Component payloads: the data Next.js fetches in the background so that a click on a link feels instant. Next.js prefetches every <Link> that enters the viewport, and Googlebot’s renderer is a viewer. The home page alone had 78 internal links. Each render of each page queued a few dozen prefetch requests, and Google crawled them like any other URL.

The responses were marked noindex and uncacheable, so none of them ever reached the index. They simply spent the budget that the guides needed.

Measuring it before touching anything

To confirm the cause we loaded four pages in a real Chromium, scrolled each one to the bottom and counted requests carrying _rsc=. On the live site the counts were 30 on the home page, 19 on a country page, 31 on the guides hub and 27 on a guide, out of 41 to 52 requests per page.

The fix

Every internal link on the site already went through one wrapper component, so the fix was one default: prefetch off. The pages are server-rendered, so a click without prefetch still lands on complete HTML; it is simply fetched at click time.

export default function SiteLink({ prefetch = false, ...props }: SiteLinkProps) {
  return <Link prefetch={prefetch} {...props} />;
}

The second half is robots.txt, for the prefetch URLs Google has already discovered. Rendering never needs them, because the HTML response carries the page and its hydration data, so blocking them does not change what Google sees:

Disallow: /*?_rsc=
Disallow: /*&_rsc=

After the deploy the same four pages made zero _rsc= requests, and total requests per page fell from 41-52 to 17-21. Client-side navigation and the back button kept working.

Should you do the same?

  • Open Search Console, Settings, Crawl stats, and look at "By file type". If "Other" dominates and its examples carry _rsc=, you have the same problem.
  • The cost shows up only when a site has many pages and little authority. A new site with hundreds of URLs and no backlinks gets a small crawl budget, and every prefetch is taken out of it.
  • Keep prefetch where it matters for people, for example a primary call to action, by passing prefetch explicitly on that one link.
  • Check the change by crawling your own pages in a real browser and counting the requests, not by trusting the setting.

The change went live on 27 September 2026. We will update this post with the Search Console numbers once Google has recrawled the site.

80% of Googlebot’s requests went to Next.js prefetches. Here is the one-line fix.