amazonbot
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
40 |
bingbot
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
40 |
proximic
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
60 |
grapeshotcrawler
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
60 |
archive.org
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
40 |
archive.org_bot
No rules defined. All paths allowed.
Other Records
Field |
Value |
crawl-delay |
40 |
Comments