sprintcourseLesson 1 of 30 · 9 min · 7 steps
Day 1. How search engines find and display a page
The key idea
Three stages: crawling, indexing and ranking
A search engine does not search the live web when you enter a query. It does three things in advance:
- Crawling. A bot follows links and sitemaps and downloads pages. If no links point to a page, the bot may never discover it.
- Indexing. The search engine analyzes the page: text, headings, links, images and markup. It then decides whether to store it in the index. Duplicate, empty and excluded pages often do not make it.
- Ranking. When someone searches, the system selects relevant pages from the index and orders them using many signals: intent, quality, usability, authority, freshness and location.
Why this matters
A page can get lost at any stage. It cannot rank before it is indexed, and cannot be indexed before it is discovered and crawled. Diagnose SEO problems in that order: access first, then indexing, then rankings.
What SEO can and cannot do
SEO makes a page easy to discover, clear about its subject and convincingly useful compared with other results. It cannot guarantee first place: competitors, the query and search algorithms all matter. Aim for more relevant traffic and leads, rather than a guaranteed number-one ranking.
A page's journey into search results
A page the crawler cannot find
A coffee shop launches a page about coffee beans for Turkish coffee, but links to it from neither the menu nor the catalog.
Before
No menu, catalog or sitemap points to the page. The crawler has no obvious way to discover its URL.
After
The page is linked from its catalog section and included in the sitemap, giving the crawler discovery paths on a future crawl.
How an expert reasons
First ask how a crawler can discover the URL. An internal link from a relevant section is a reliable path; the sitemap is an additional signal. Submitting a URL in webmaster tools can help discovery but does not replace links or guarantee indexing.
An unrealistic promise
Treat a guarantee of a top-three ranking in one month with caution. Search engines determine rankings. A sound commitment is: remove obstacles, improve pages to meet demand, and agree on how results will be measured.
Check your understanding
Check your understanding
See your site through a crawler's eyes
- Search Google or Bing for
site:yourdomain.example. Roughly how many pages appear? - Compare this with the number of useful product, service and article pages on your site.
- Pick an important page and search for
site:yourdomain.example/page-path. Does it appear?
In a note, record the approximate indexed count, your actual page count and one missing page. The site: operator gives an incomplete approximation; day 26 covers more reliable Search Console reports.
Saved automatically, even offline.