99creatix Book a 15-minute fit call
Glossary · SEO basics

crawling

Also called web crawler, Googlebot

Reviewed 31 Oct 2026 By Madiha Fatima 4 min read Check it yourself: yes, free

4 min
Definition

Crawling is how a search engine discovers the pages on your website. A program called a crawler (Google's is named Googlebot) starts from addresses it knows, reads each page and follows every link it finds to the next one. A page that nothing links to and no sitemap lists may never be crawled, so it never reaches the index.

Crawling is not the same as ranking or even indexing. Google can crawl a page and decide not to keep it. But nothing good happens to a page that is never crawled, which is why an SEO audit starts by asking whether Google can reach every page that matters and is not wasting its time on pages that do not.

Short version: if Google cannot reach the page, nothing else about it matters.
Why it matters for your business

Google only ranks
what it has read.

A real estate agency with four hundred listings expects them all to be in Google. If the listings load only through a search box or a filter, there is no link for a crawler to follow and most of them are invisible. Nobody did anything wrong. The site was built for people and the crawler is not a person.

The other common waste is the reverse: a shop whose colour, size and sort filters create tens of thousands of addresses for the same forty products. Google spends its visit crawling filter combinations and gets to the new product page days later than it should. Robots.txt and canonical tags exist for exactly this.

When a proposal lists 'crawl optimisation' or 'crawl budget', this is what it means. On a site under a few thousand pages it is usually a one-off fix, not a monthly line.

Example

Four hundred listings,
sixty of them crawlable.

What the agency sees

Every property is on the site. You pick an area and a price range, press Search and the listings appear. New properties go up the same day.

What Googlebot sees

A search form and a map. The listings are fetched when a visitor presses the button, so the crawler never sees a link to any of them. The sixty that are crawled are the ones featured on the homepage. Adding a plain 'All properties' page with a link to every listing, split by area, and listing those addresses in the sitemap made the rest reachable. Enquiries from Google rose because pages that had never been crawled were finally in the index, not because any ranking changed.

How to check it on your site

Four checks on whether Google can get around your site.

Tick each one as you go. 0 of 4 checked on your site.

  1. Count what Google has found

    Search site:yourdomain.com and compare the number with the pages you expect. In Search Console, the Pages report splits the rest by reason. 'Discovered, currently not indexed' means found but not yet crawled. 'Crawled, currently not indexed' means crawled and dropped.

  2. Click through your own site with scripts off

    Turn off JavaScript in your browser (in Chrome: Settings, Site settings, JavaScript) and reload your site. Anything that disappears, such as a menu, a listings grid or a product list, is content a crawler may struggle to reach. Turn it back on afterwards.

  3. Read robots.txt

    Open yourdomain.com/robots.txt. A line reading Disallow: / blocks the whole site, and we see it on live sites that were copied from staging. Blocking your cart or admin area is fine. Blocking your images, scripts or any folder that holds real pages is not.

  4. Look at the crawl stats

    In Search Console, under Settings, open Crawl stats. A healthy site shows steady daily requests and mostly 200 responses. A spike in server errors or a response time above a second means your host is slowing the crawler down, and it will visit less often.

Related service

Crawl problems are found in the technical audit in month one of the SEO service and designed out of every site we build. Prices are on the price list, not on a call.

See the SEO service
Reviewed by Madiha Fatima, founder of 99creatix

Every entry in this glossary is written and checked by the person who runs the agency, reviewed at least once a quarter and dated. Last reviewed 31 October 2026.

Spotted an error or a term we are missing? Tell us and it is fixed within the week.

Did this explain it?
Next step

Not sure Google can
reach all of your pages?

Fifteen minutes, your website and Search Console. We will show you which pages Google has found, which it has not and what is in the way, before you decide anything.