Indexing
Indexing is the process by which search engines store crawled web pages in a searchable database so they can be retrieved as results — a page must be indexed before it can rank or be cited.
Also known as: search indexing, index inclusion, page indexation
Indexing is the process by which search engines store and organize crawled web pages in a searchable database. A page must be indexed before it can appear in search results or be cited in AI answers built on top of search. Indexing is distinct from crawling — crawling fetches the page, while indexing decides whether to add it to the searchable database and how to represent it there.
What Indexing Means
Indexing is the database step search engines run after crawling. It evaluates each crawled page for inclusion, checking quality signals, duplicate detection, and content value, then stores accepted pages in the searchable index that powers query responses. Pages can be excluded at multiple points: blocked by robots, noindexed by directive, deemed duplicates of canonical alternatives, or judged too thin or low-value to merit a slot. Search console reports the status of each tracked URL — indexed, excluded, with reasons — which is the authoritative source for what's actually in the index versus only crawled.
How Indexing Works
Indexing works through a sequence: a crawler fetches a page, the rendering engine processes any JavaScript needed to see the content, the indexing system evaluates the page for inclusion (checking quality signals, duplicate detection, content value), and an indexed page becomes eligible to rank for relevant queries. Submission through search console's URL Inspection tool can accelerate indexing for individual pages, and inclusion in the XML sitemap signals priority. Mobile-first indexing means Google primarily uses the mobile version of a site for indexing and ranking, so the mobile version is the source of truth for content, links, and metadata.
Common Pitfalls and Misconceptions
A common mistake is assuming every page on a site is automatically indexed. Modern search engines are selective: a fully crawlable page can still be excluded from the index if quality signals don't meet a threshold, if the content duplicates other indexed pages, or if the engine simply decides it doesn't need to keep that page. Another misconception is that indexing and crawling are the same thing. They are sequential — a page must be crawled before it can be indexed, but being crawled doesn't guarantee indexing. The 'Crawled but not indexed' status in search console is one of the most common quality-based exclusions and signals that the page hasn't earned an index slot.
Indexing in Practice
The mature practice is to monitor indexation as a portfolio metric, not just at the individual-page level. Track the share of important URLs that are indexed, the patterns of exclusions ('Crawled — currently not indexed' versus 'Discovered — currently not indexed' versus 'Duplicate without user-selected canonical'), and the trend over time. Sites that watch indexation as a system metric catch quality and architecture problems early — usually months before the symptoms show up in traffic — and they make different decisions about which pages to invest in versus consolidate or remove. AI answer engines built on search indexes only cite pages in those indexes.
Frequently asked questions
-
What is the difference between crawling and indexing?
Crawling is the process of search bots fetching pages from the web. Indexing is the subsequent process of evaluating those pages and storing them in a searchable database. A page must be crawled before it can be indexed, but being crawled doesn't guarantee indexing — engines actively exclude pages they judge low-value, duplicate, or otherwise unsuitable.
-
How do you check if a page is indexed?
Use Google Search Console's URL Inspection tool for authoritative status, or run a site: search (site:example.com/specific-page) for a quick check. The URL Inspection tool also explains why a page isn't indexed when that's the case — a noindex directive, robots block, canonical to another URL, or a quality-based exclusion.
-
Why do pages get excluded from the index?
Common reasons include noindex directives, robots.txt blocks, duplicate content consolidated under a different canonical, soft 404s, redirects to other URLs, low quality or thin content, and 'discovered but not indexed' or 'crawled but not indexed' decisions by the engine. The last two are quality-based exclusions and signal that the page hasn't earned an index slot.
-
How do you get a page indexed faster?
Submit the URL through search console's URL Inspection, add the page to the XML sitemap, link to it from other indexed pages on the site (especially the homepage or hub pages), and ensure the content meets quality and relevance expectations. New pages are typically indexed within hours to a few days when crawlability and quality are strong.
-
What is mobile-first indexing?
Mobile-first indexing means Google primarily uses the mobile version of a site for indexing and ranking. The mobile version is the source of truth for content, links, and metadata, even when queries come from desktop users. Sites with weaker mobile experiences than desktop ones — fewer features, less content — typically suffer indexing and ranking consequences.
-
Can a page be partially indexed?
Not as a formal status, but in practice yes. A page can be indexed without all of its content (if JavaScript rendering fails and content depends on it), without rich features (if structured data has errors), or without certain variations (if hreflang is broken). Indexed-but-not-fully-recognized is a common state worth diagnosing.
-
How does indexation relate to AI answer engines?
AI answer engines that build on search indexes (Google's AI Overviews, Bing's Copilot) only cite pages that exist in those indexes — indexation is the prerequisite for AI visibility on those surfaces. Standalone AI tools with their own crawl may have different inclusion criteria, but the same fundamentals apply: crawlable, high-quality, well-structured pages are eligible for citation in both worlds.