bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
90 |
archive.org_bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
90 |
ia_archiver
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
90 |
ia_archiver-web.archive.org
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
90 |
facebookexternalhit
twitterbot
fyyd-poll
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
280 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
360 |
dotbot
r6_feedfetcher
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
600 |
metauri
mediawords
exabot
exabot
mojeekbot
trendictionbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
900 |
gigabot
megaindex.ru
megaindex
ccbot
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3000 |
seznambot
jobboersebot
ahrefsbot
rogerbot
mj12bot
spbot
simplepie
seoscanners.net
dotbot
yandexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5000 |
vagabondo
kraken
semrushbot
blexbot
bubing
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20000 |