clickagy*
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
ccbot*
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-image
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-video
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-news
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-favo
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-cpro
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider-ads
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
google*
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
yandex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |