Robots Exclusion Standard

/robots.txt (spec)

An informal standard that allows sites to ask web crawlers to not crawl certain urls.

• Search robots.txt
• Parse robots.txt