Robots Exclusion Standard

/robots.txt (spec)

An informal standard that allows sites to ask web crawlers to not crawl certain urls.