googlebot
googlebot-news
googlebot-image/1.0
googlebot-video/1.0
googlebot-mobile
adsbot-google
googleweblight
google-searchbyimage
bingbot
bingpreview
msnbot
msnbot-media
adidxbot
yandex bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider
baiduspider-image
baiduspider-mobile
baiduspider-news
baiduspider-video
facebot
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
duckduckbot
exabot
ia_archiver
No rules defined. All paths allowed.