*
| Rule |
Path |
| Disallow |
/members/ |
| Disallow |
*?s= |
| Disallow |
*p%3D*%26preview%3Dtrue* |
| Disallow |
/wp-content/cache/ |
| Disallow |
/wp-comments-post.php |
| Allow |
/wp-content/cache/autoptimize/ |
| Disallow |
/docs/ |
| Disallow |
/detroitchicago/ |
| Disallow |
/porpoiseant/ |
| Disallow |
/edmontonalberta/ |
| Disallow |
/api/ |
| Disallow |
/businesses-for-sale/?encodedurl=* |
twitterbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
facebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yahoo! slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexmedia
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandeximages
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexcatalog
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yadirectfetcher
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexblogs
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexnews
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexpagechecker
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexmetrika
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexmarket
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
yandexcalendar
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
piplbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
sentibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
qwantify
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
neevabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
grapeshot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
siteauditbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
semrushbot-ba
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
semrushbot-si
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
semrushbot-swa
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
semrushbot-ct
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
splitsignalbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
semrushbot-coub
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
seekportbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
awariorssbot
awariosmartbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
turnitinbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
youbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
amazonbot
| Rule |
Path |
| Disallow |
*.htmlemail/ |
| Disallow |
*.htmlemail/* |
| Disallow |
/?p=* |
| Disallow |
*.html/feed* |
| Disallow |
*.html/email/ |
| Disallow |
*.html/email/* |
Other Records
| Field |
Value |
| crawl-delay |
60 |
dataforseobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |