Discovery and crawling
Search engines find URLs through links, sitemaps, prior crawls, and other signals. Technical barriers, weak internal linking, duplicate paths, and unstable responses can waste crawl activity or keep important pages out of reach.
