Crawling is how a search engine discovers the pages on your website. A program called a crawler (Google's is named Googlebot) starts from addresses it knows, reads each page and follows every link it finds to the next one. A page that nothing links to and no sitemap lists may never be crawled, so it never reaches the index.
Crawling is not the same as ranking or even indexing. Google can crawl a page and decide not to keep it. But nothing good happens to a page that is never crawled, which is why an SEO audit starts by asking whether Google can reach every page that matters and is not wasting its time on pages that do not.
Crawling is URL discovery and fetching by Googlebot and other bots under the rules in robots.txt, within a crawl budget shaped by host capacity and perceived demand. Discovery comes from internal links, the XML sitemap, external links and previously known URLs. Pages that depend on client-side rendering are fetched, queued for rendering and processed later.
Common crawl problems: links only in JavaScript-built menus, faceted navigation generating unbounded URL spaces, redirect chains, slow server response and robots.txt rules copied from staging. The Crawl stats report in Search Console (under Settings) shows requests, response codes and host status. The Pages report shows URLs discovered but not crawled.
User-agent: *
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml 