Find what's keeping your site invisible to Google.

Googlebot has to find you first.

Think of the site as a graph: URLs are the nodes, internal links the edges between them. A page with no incoming link, or one buried under three layers of session-ID noise, never enters Google's queue at all. The first file Googlebot reads on every visit is /robots.txt. Those few lines decide which URLs it crawls and which it skips.

# robots.txt for apparelclient.com
User-agent: Googlebot
Disallow: /cart
Disallow: /*?sort=
Disallow: /*?filter.size=
Allow:    /collections/

Sitemap: https://apparelclient.com/sitemap.xml

One wrong line here can delist a whole site in minutes, so it's where I start every audit.

Your JavaScript sits in a queue, sometimes for weeks.

The cheap pass is HTML. It gets fetched, parsed, and indexed in seconds. The expensive pass renders your app in a headless Chromium called the Web Rendering Service. That queue usually clears within a day. I've watched it run eleven. If your title and body only appear after JS executes, you wait, and Google doesn't publish how long the wait is.

// abridged WRS timing on a real client
06:14:02  /pricing             [HTML crawled]
06:14:02  /pricing             [render queued]
14:32:17  /pricing             [WRS render]
14:32:17  /pricing             [indexed in 8h 18m]

That log is one page on one client. Search Console never surfaces this queue, so JS-heavy sites can sit un-indexed for days before anyone notices.

A search is an intersection, not a scan.

When you search, Google doesn't read the web. It reads an inverted index it built ahead of time. Each word maps to a postings list, just the documents that contain it. A query then resolves by intersecting a handful of those short lists instead of scanning trillions of pages. Pandu Nayak described exactly this under oath in the 2023 DOJ trial.

Query flat feet shoes → intersect the three postings → only D3 contains all three, so D3 is the candidate set ranking starts from.

The candidate set gets scored, then reranked.

BM25 ranks the candidate set by how strongly each query term matches each document. Then the neural systems take over. RankBrain, BERT, and RankEmbed project the query into vector space and pull in documents that are semantically close even when they share zero keywords. That's why "apex predator" can rank for "consumer at the top of a food chain".

// classical BM25, still load-bearing in 2026
score = idf · tf(k₁+1) / (tf + k₁(1 − b + b·|d|/avgdl))

idf("flat")    = 1.58
tf in D3       = 2
|d| (D3 length) = 8
avgdl          = 7.3

score(D3) = 3.24

Numbers are illustrative, but the BM25 math is real and hasn't changed since 1995.

Real users re-rank you, live.

Confirmed in the 2023 DOJ trial: NavBoost uses 13 months of click data to reorder results. A long dwell after the click lifts you. Pogo-stick straight back to the SERP and try the next result, and that counts against you. Your CTR is a live input the ranking system actually reads.

// NavBoost click signal aggregation, 13mo window
month 01 · R1  user → R1 → ← back in 3s          badClick
month 02 · R3  user → R3 → dwell 4m12s          goodClick
month 03 · R3  user → R3 → converted             signal++
month 05 · R2  user → R2 → pogo-stick            badClick
month 13     aggregate · R3 dominates         → rerank

A weak snippet gets demoted no matter how clean the rest of the stack is.

Twiddlers do the final reordering.

Internally, Google calls them twiddlers, lightweight rerankers that fire after the core ranking is done. Freshness nudges one way, diversity caps you at one result per host, spam demotions pull the rest down. By the time a result lands on page one, a stack of these has already adjusted its position.

Four twiddlers, each nudging the same result after core ranking is supposedly done.