femtosearchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
200 |
mappy
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
200 |
um-fc/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
um-ic/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
um-ln/1.0
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
linguee
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
100 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
buck
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
yandexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2.0 |
yandeximages
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2.0 |
yandex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2.0 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
40 |
applebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
30 |
yahoo! slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
40 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
100 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
300 |
amazonadbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
90 |
ccbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
zoombot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |