*
| Rule |
Path |
| Disallow |
/engine/ajax |
| Disallow |
/sms |
| Disallow |
/engine/go.php |
| Disallow |
/user/ |
| Disallow |
/newposts/ |
| Disallow |
/statistics.html |
| Disallow |
/*subaction%3Duserinfo |
| Disallow |
/*subaction%3Dnewposts |
| Disallow |
/*do%3Dlastcomments |
| Disallow |
/*do%3Dfeedback |
| Disallow |
/*do%3Dregister |
| Disallow |
/*do%3Dlostpassword |
| Disallow |
/*do%3Daddnews |
| Disallow |
/*do%3Dstats |
| Disallow |
/*do%3Dpm |
| Disallow |
/currency |
| Disallow |
/goroskop |
| Disallow |
/gsearch |
| Disallow |
/g1search |
| Disallow |
/g2search |
| Disallow |
/ysearch |
| Disallow |
/*prikoli |
| Disallow |
/*games |
| Disallow |
/*print |
| Disallow |
/*action%3Dmobile |
| Disallow |
*/page/* |
Other Records
| Field |
Value |
| crawl-delay |
120 |
yandex
| Rule |
Path |
| Disallow |
/*prikoli |
| Disallow |
/*games |
| Disallow |
/goroskop |
| Disallow |
/gsearch |
| Disallow |
/*do%3D |
| Disallow |
/user |
| Disallow |
/*print |
| Allow |
*/page/* |
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot-mobile
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
adsbot-google-mobile
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
adsbot-google
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
tineye-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
qwantify
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
exabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
coccoc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
umbot-ln
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
queryseekerspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sogou spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sogou web spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
youdaobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
spbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
linkdex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
linkdexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
semrushbot-sa
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
cfnetwork
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
sputnikbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
turnitinbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
proximic
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |
grapeshot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
500 |