*
| Rule |
Path |
| Disallow |
/cgi-bin/ |
| Disallow |
/CSS/ |
| Disallow |
/js/ |
| Disallow |
/_notes/ |
| Disallow |
/buttons/ |
| Disallow |
/catalogs/ |
| Disallow |
/cbox/ |
| Disallow |
/data/ |
| Disallow |
/epubs/ |
| Disallow |
/images/ |
| Disallow |
/magiczoomplus/ |
| Disallow |
/newsite/ |
| Disallow |
/ouijs/ |
| Disallow |
/picturelibrary/ |
| Disallow |
/Merchant2/admin.mvc |
| Disallow |
/Merchant2/epubs/ |
bingbot
| Rule |
Path |
| Disallow |
/Merchant2/graphics |
Other Records
| Field |
Value |
| crawl-delay |
90 |
yandex
| Rule |
Path |
| Disallow |
/Merchant2/graphics |
Other Records
| Field |
Value |
| crawl-delay |
500 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yahoo! slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
msnbot-media
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
twitterbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
tweetedtimes bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
tweetmemebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
paperlibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
tineye-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
qwantify
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
exabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
coccoc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
umbot-ln
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
queryseekerspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sogou spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sogou web spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
youdaobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
spbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
linkdex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
linkdexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
cfnetwork
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sputnikbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
turnitinbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
proximic
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
30 |
grapeshot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
30 |