Well-Known is a free, open index of well-known resources. It scans millions of domains at regular intervals, compiling a searchable database of structured well-known resource data.
The search engine supports specific search filters for each resource type, enabling detailed queries to answer a wide variety of questions.
8 resources are currently supported.
Well-known resources are files hosted by sites at specific "well known" locations. These resources are used to communicate a variety of information about the site and the services it provides to developers, other sites and services, etc.
Resource formats and locations are usually defined in specifications, though some have
arisen from loose consensus without being formally specified. They are often located
You can read more about well-known resources here.
The well-known resource scans are performed by WellKnownBot. All requests contain a
User-Agent header that includes the string
the URL of this page.
While WellKnownBot makes automated requests, it is not a web crawler/spider. It only ever requests a small number of specific resources that are intended for public, programmatic consumption.
Because it is not a crawler, WellKnownBot does not follow generic
User-Agent: * crawling rules in robots.txt files. However, it does obey
robots.txt rules in groups that specifically target it.
If you wish to restrict the resources WellKnownBot scans, add a
User-Agent: WellKnownBot group to your site's robots.txt.