*
| Rule |
Path |
| Disallow |
/cgi-bin |
| Disallow |
/wp-content/plugins/ |
| Disallow |
/wp-content/themes/ |
| Disallow |
/wp-includes/ |
| Disallow |
/*/attachment/ |
| Disallow |
/tag/*/page/ |
| Disallow |
/tag/*/feed/ |
| Disallow |
/page/ |
| Disallow |
/comments/ |
| Disallow |
/xmlrpc.php |
| Disallow |
/?attachment_id* |
| Allow |
/wp-content/uploads/* |
| Allow |
/wp-content/*.js |
| Allow |
/wp-content/*.css |
| Allow |
/wp-includes/*.js |
| Allow |
/wp-includes/*.css |
| Disallow |
/*? |
*
| Rule |
Path |
| Disallow |
/?s= |
| Disallow |
/search |
*
| Rule |
Path |
| Disallow |
/trackback |
| Disallow |
/*trackback |
| Disallow |
/*trackback* |
| Disallow |
/*/trackback |
*
| Rule |
Path |
| Allow |
/feed/$ |
| Disallow |
/feed/ |
| Disallow |
/comments/feed/ |
| Disallow |
/*/feed/$ |
| Disallow |
/*/feed/rss/$ |
| Disallow |
/*/trackback/$ |
| Disallow |
/*/*/feed/$ |
| Disallow |
/*/*/feed/rss/$ |
| Disallow |
/*/*/trackback/$ |
| Disallow |
/*/*/*/feed/$ |
| Disallow |
/*/*/*/feed/rss/$ |
| Disallow |
/*/*/*/trackback/$ |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
60 |
noxtrumbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
20 |
msiecrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
webcopier
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
httrack
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
microsoft.url.control
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
libwww
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
orthogaffe
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
ubicrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
doc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
zao
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
sitecheck.internetseer.com
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
zealbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
msiecrawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
sitesnagger
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
webstripper
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
webcopier
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
fetch
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
offline explorer
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
teleport
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
teleportpro
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
webzip
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
linko
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
httrack
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
microsoft.url.control
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
xenu
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
larbin
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
libwww
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
zyborg
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
download ninja
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
wget
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
grub-client
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
k2spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
npbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
webreaper
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
120 |
googlebot
| Rule |
Path |
| Allow |
/*.css$ |
| Allow |
/*.js$ |
Comments