Faceted navigation is how a shopper narrows a catalogue. It is also, according to Google, the most common source of overcrawl that site owners report. Each extra filter is not one URL. It is every previous combination multiplied by the new value. Colour times size times brand times sort times page number is how a 400-product storefront presents as a million-URL site.
The SEO mistake is to treat that explosion as one switch: index everything, or block everything. Index everything and you drown the crawler in near-duplicates. Block everything and you throw away the filtered views that match how people actually search ("red running shoes", "oak dining tables", "size 10 boots"). The work is triage per facet, then controls that match the triage.
What Goes Wrong When Filters Mint URLs
Googlebot discovers links. Facet links are often ordinary <a href> elements, which means every crawl of a category discovers a new set of combinations. Those combinations link to more combinations. Discovery never finishes. New product pages wait in line behind ?colour=navy&size=12&sort=price-asc&page=7.
Index bloat follows. Search Console fills with duplicate variants, crawled-currently-not-indexed, and soft 404s on zero-result combinations. Ranking signals split between the clean category and a dozen filtered copies. Titles and canonicals start to disagree because the platform generated them from the request URL.
There is a server cost as well. Filtered views are expensive to render. Google has been explicit that crawling faceted URLs tends to consume large amounts of computing resource, and that this crawl delays discovery of content you do care about. Crawl budget is not a vanity metric on a catalogue. It is whether today's new SKU gets fetched this week.
Google's Two Roads
In December 2024 Google restated the choice in plain language.
If you do not need faceted URLs in search, prevent crawling. robots.txt disallow on the filter parameters is the method they put first. URL fragments (#colour=red) are the other: Google Search generally does not treat fragments as separate crawlable URLs, so the filter state stays client-side.
If you do need some faceted URLs crawled, follow their hygiene rules. Use the standard & separator, not creative delimiters. Keep filter order stable in the path or query so the same selection cannot appear as two URLs. Return 404 for combinations with no results. Do not redirect empty results to a generic not-found document if you can avoid it. Canonicals and rel="nofollow" on filter links are listed as weaker, slower methods.
That order of preference is the opposite of a lot of legacy SEO advice that started with noindex, follow on every facet. noindex still requires a crawl to read the tag. On a combinatorial URL space you have invited Googlebot to download the haystack in order to be told to drop each straw. Google's large-site crawl guidance is blunt on the same point: do not use noindex as a crawl-budget tool.
Triage Facets Before You Touch Robots
We split filters into buckets with one question: does this view match a search people actually make, and can we support that view with unique, useful content?
Index and support: a single filter (sometimes two) with real query demand, inventory behind it, and a page that can carry its own title, copy, and internal links. Brand plus category is the usual example. Material or size only when the query data says people search that way and the page is more than a checkbox state.
Keep for users, keep out of the index: useful for shoppers, useless as a landing page. Multi-filter combinations, sort orders, "in stock" toggles, price sliders. These should not be discovered as infinite HTML links if we can help it.
Do not crawl: parameters that only change sort, view, session, or tracking. Session IDs. utm leftovers. Facets whose values create near-duplicate lists.
404: combinations with zero results. An empty grid with a 200 status becomes a soft 404 magnet and still costs a crawl.
Canonical-only: a filtered URL that is genuinely the same list as the parent, where the filter does not change the set in a meaningful way. Canonical to the parent is a consolidation hint. It does not, by itself, stop the crawl. Pair it with a crawl control if volume is the problem.
Ecommerce SEO only pays for the first bucket when the other buckets are contained. A beautiful brand facet page cannot rank if Googlebot never reaches it because it is busy on sort parameters.
Controls That Match The Bucket
Robots.txt for the combinations you never want fetched. Be precise. A broad Disallow: /*?* will block useful pagination, search, or the one parameter you meant to keep. Disallow the named parameters (color, size, sort, dir) and allow the exceptions you have decided to index. Test with the robots tester and with live fetches. Staging and production often drift.
Fragments or client-side filters for pure UI state. This is the cleanest crawl control because extra URLs never exist. It is also a product decision: shareable filtered links go away unless you add an explicit "link to this view" that creates a single, approved URL.
Canonicals on variants that must remain reachable. Point at the approved landing URL, not at whatever the request produced. Do not canonicalise a substantially different list to the parent and expect Google to obey. The hint is strongest when the content is actually a duplicate.
rel="nofollow" on filter links only works if every link to those URLs carries it, internal and external. Google describes this as weaker than robots.txt. We use it as a supplement inside a page (do not paint every checkbox as a followed crawl path), not as the main lock.
Noindex on a crawlable URL when something is already in the index and must come out, and you still need the crawler to see the instruction. Once the URL has dropped, stop paying the crawl tax: move that pattern to robots.txt. Never combine noindex and robots.txt disallow on the same URL. If Googlebot cannot fetch the page, it cannot see the noindex, and a previously indexed URL can sit in the index for a long time.
Sitemaps list only the URLs you want indexed. Filtered URLs that "might be useful" do not belong there. Sitemaps are a preference signal and a discovery path. Do not advertise the haystack.
Internal links from blog posts, navigation, and footer should name the approved facet landings, not a leftover query string from a merchandising widget.
URL Hygiene If A Facet Must Rank
Stable parameter order. ?brand=acme&colour=red and ?colour=red&brand=acme are two URLs unless you normalise. Pick an order, enforce it in the application, 301 the other order if it already leaked.
One encoding. Hyphens versus underscores, encoded commas versus repeated keys, uppercase versus lowercase values: pick one. Redirect the rest.
No empty values, no duplicate keys, no leftover page=1 on the first page. page=1 is a duplicate of the clean URL.
404 on zero results, as Google asked. A 200 empty state with a self-canonical is how you manufacture soft 404s.
Do not build facet landings that are only a filtered grid. If the view is allowed to rank, it needs a unique title, a short unique introduction that is true of that subset, and a way back to the parent category. Thin generated copy across a thousand facets is how you trade index bloat for thin-content bloat.
How We Validate After The Change
Crawl the category tree with parameters allowed, then again with production robots.txt respected. The second crawl should be dramatically smaller. Log files should show Googlebot leaving the disallowed patterns alone after it has recached robots.txt.
Search Console should, over a lag, show fewer duplicate and crawled-not-indexed URLs in the faceted patterns. URL Inspection on an approved facet landing should show it as indexable, with the self-canonical you expect. Inspection on a blocked combination should show robots.txt as the reason.
Watch the first weeks after a robots.txt change. If you disallow URLs that were ranking, those queries can drop until the approved landing inherits them, if it can. That is why triage happens before the disallow, not after a panic report.
Rendering still matters. A filter implemented only in JavaScript, with no crawlable link, may already be invisible to Googlebot. Do not "fix" that by suddenly exposing every combination as HTML unless you have the buckets ready.
Working Order
Inventory the parameters. Map each to a bucket. Implement 404s on empty combinations. Normalise URL order. Put approved landings in the sitemap and in navigation. Then robots.txt the rest. Re-crawl. Only then hunt leftovers with noindex.
If the catalogue is still growing faster than the rules, the rules are not tight enough, or merchandising keeps adding followed links from widgets. We would rather tighten discovery than write 400 unique facet essays. Send the category tree and a Search Console export if crawl is stuck on filters and you need the buckets decided against live query data.
