Robots.txt: Understanding the Basics of Crawler Management

Robots.txt allows you to suggest which pages each search engine should not crawl. Learn the basics in this blog post.