← BlogTech case · SEO

The Sitemap Google Couldn’t Read

Auth middleware quietly gated the one file whose entire job is telling Google the site exists. The confusing part wasn’t the bug — it was that the page looked perfect in my own browser, and that fixing it changed nothing for weeks.

August 28, 2026·6 min read·Next.js · Clerk · SEO · Middleware

302what Googlebot got instead of XML

2independent gates that both have to let a crawler through

weeksa cached block can outlive the fix that solved it

Google Search Console was reporting pages as "URL is unknown to Google" — with no referring sitemap. Which is a strange thing to read when you have a sitemap, you submitted it, and you can open it in your browser right now and see every URL sitting there in well-formed XML.

That last part is the trap, and it took me longer than I'd like to admit to see it. I was checking the sitemap the way a person checks a webpage: by opening it. In a browser. Where I was logged in.

Googlebot is not logged in. And in a Next.js app with auth middleware in front of it, that difference decides whether your site gets indexed at all.

Fix 01

Your sitemap.xml is not a file

This is the whole misconception the bug grows out of. In the App Router, sitemap.xmlusually isn't a file sitting on disk — it's generated by src/app/sitemap.ts, which Next.js turns into a route. It runs code. It hits your database. And, like every other route in your app, it passes through middleware on the way out.

Meanwhile robots.txt in /public genuinely is a file, served straight off the static file server, and it never touches middleware at all.

So you end up in the worst possible diagnostic position: the two files whose job is to talk to crawlers behave in completely opposite ways, and nothing in their names or their location suggests it. Your robots.txt is always fine. Your sitemap may be behind a login wall. Both look identical from your browser.

Same two "files", two different paths through the stack

robots.txt

static file server

200 — never sees middleware

sitemap.xml

middleware

not a public route → 302 to login

What Googlebot received was a redirect to an accounts subdomain. Not an error, not a 404 — a perfectly polite 302pointing at a sign-in page. From Google's side there was simply no sitemap, so the pages it listed were never queued for discovery, so Search Console truthfully reported that it had never heard of them.

Fix 02

Two gates, and knowing which one you're fixing

Clerk's middleware has two separate mechanisms that decide a request's fate, and they are easy to confuse because both look like "a list of paths":

→The matcher (exported config) decides whether middleware runs at all for a request.
→The public-route matcher decides what middleware does once it has already been invoked.

You can solve the sitemap problem at either level, and it's worth understanding which one you picked. Adding the path to the public-route list lets middleware run and then wave the request through. Excluding the extension in the matcher means middleware never executes for that request in the first place.

This app ends up doing both, for different reasons — and the second one exists because of a failure mode I hadn't anticipated: without excluding .xml and .txt from the matcher, middleware intercepts requests for files Vercel is supposed to serve out of /public and hands back a 404 instead.

src/proxy.ts — gate 1: what middleware does when it runs

const isPublicRoute = createRouteMatcher([
  '/',
  '/sitemap.xml',
  '/sitemap(.*).xml',   // covers split sitemaps
  '/blog(.*)',
  '/quiz(.*)',
  '/learn(.*)',
  // ...
])

export default clerkMiddleware(async (auth, req) => {
  if (!isPublicRoute(req)) {
    await auth.protect()
  }
})

src/proxy.ts — gate 2: whether middleware runs at all

export const config = {
  matcher: [
    // Skip Next.js internals and static assets — but NOT .html (bots probe .html paths).
    // `.txt` (robots.txt) and `.xml` (sitemap.xml) must be excluded so Vercel can
    // serve them from /public without Clerk intercepting and returning 404.
    '/((?!_next|[^?]*\\.(?:css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest|txt|xml)).*)',
    '/(api|trpc)(.*)',
  ],
}

Note the deliberate omission in that regex: .html is not in the skip list. Bots probe .html paths constantly looking for stale files and admin panels, and those requests should keep going through the middleware that handles them.

Fix 03

The fix that looks like it didn't work

Here's the part that turns a twenty-minute bug into a multi-week one. You deploy the fix. You verify it. You go back to Search Console — and it still says the pages are unknown. Still no referring sitemap. Nothing has changed.

Because the crawler's memory is part of your system, and you just changed the server without changing the memory. Google cached the answer it got last time. On a low-traffic domain with no particular reason to re-crawl aggressively, that cached "this is blocked" verdict can sit there for weeks after the block is gone.

A fix that only exists on your server is half a fix. The other half is convincing the crawler to come back and look again.

The other half is manual: URL Inspection in Search Console, then Request Indexing, per URL. Resubmitting the sitemap helps it re-discover the list, but for pages already marked as blocked, the explicit request is what forces a fresh fetch. Don't wait for organic re-discovery to prove your fix worked — it will take longer than your patience, and the whole time you'll be wondering whether you fixed the right thing.

Fix 04

The same bug wearing a different hat

A few months later I shared the site in a LinkedIn message and the link preview came back with a title, a domain, and an empty grey rectangle where the image should be.

Different platform, different crawler, and this time the server was genuinely blameless. I checked it properly instead of guessing: the og:image tag was present with an absolute URL, the image route returned 200 with image/pngat 1200×630 in under half a second, there were no redirects, robots.txt didn't block it, and requesting it with LinkedIn's own user agent returned exactly the same bytes as a browser.

Every one of those checks passing is what told me where the problem actually lived. Same lesson as the sitemap, in a new costume: the crawler had cached a preview from an earlier visit — from before that image existed — and a new share doesn't trigger a re-fetch. LinkedIn keeps link previews for days, and the fix is the same shape as Google's: their Post Inspector tool, which forces a re-scrape.

Fix 05

Check it the way a crawler would

The habit worth stealing from all of this is small: when the audience for a page is a robot, stop verifying it as a human. Your browser carries a session cookie that a crawler will never have, and that one difference hides the entire class of bug.

verifying as a bot, not as yourself

# Does the sitemap actually return XML to someone with no session?
curl -sI https://yoursite.com/sitemap.xml
#   look for: 200, content-type: application/xml
#   a 302 to your login domain is the bug

# Same request, as the crawler
curl -sI -A "Googlebot/2.1 (+http://www.google.com/bot.html)" \
  https://yoursite.com/sitemap.xml

# Is the OG image reachable, and fast enough for a crawler's patience?
curl -sL -o /dev/null -w "status=%{http_code} type=%{content_type} time=%{time_total}s\n" \
  -A "LinkedInBot/1.0" https://yoursite.com/opengraph-image

One adjacent thing the sitemap route taught me while I was in there, worth a sentence because it fails just as quietly: don't stamp lastModified: new Date()on your static routes. It regenerates on every build, so every deploy tells Google that every page changed. Do that often enough and Google stops trusting the field and ignores it entirely — you've spent the signal without noticing.

src/app/sitemap.ts

// Stable lastmod for static routes — bump only when the page itself changes.
// Using new Date() on every build makes Google start ignoring lastmod.
const STATIC_LASTMOD = new Date('2026-04-27')

What I'd check first next time

→Every App Router metadata route — sitemap.ts, robots.ts, opengraph-image.tsx, RSS — is a route, not a file. If you have auth middleware, each one is a candidate for being accidentally protected.
→Know which of your two gates you fixed: whether middleware runs at all (the matcher) is a different question from what it does when it runs (the public-route list).
→Anything in /public bypasses middleware entirely — which is exactly why robots.txt looking fine tells you nothing about sitemap.xml.
→Verify with curl and a bot user agent. Your logged-in browser is the one client in the world guaranteed not to reproduce this bug.
→After the deploy, go force a re-crawl. Search Console for Google, Post Inspector for LinkedIn. A cached verdict outlives the thing that caused it.
→Keep lastmod honest. A timestamp that changes on every build is a timestamp Google learns to ignore.

None of this shows up in an error log. The app was healthy the whole time, the sitemap was valid the whole time, and every check I ran as myself came back green. The failure only existed for clients without a session — which happens to be every client that decides whether anyone finds you.