*
| Rule |
Path |
| Disallow |
/openx* |
| Disallow |
/email-alert* |
| Disallow |
/resetting* |
| Disallow |
/register* |
| Disallow |
/login* |
| Disallow |
/jobs?title=* |
| Disallow |
/my-jobs/* |
| Disallow |
/connect/* |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
amazonaws
Other Records
| Field |
Value |
| crawl-delay |
3600 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
blexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yandex
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
megaindex.ru
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
seekportbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
freshbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dataforseobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
petalbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
applebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gptbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
amazonbot
Other Records
| Field |
Value |
| crawl-delay |
3600 |
claudebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bytespider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
facebookexternalhit
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
zoombot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dataforseobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
timpibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
oai-searchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
chatgpt-user
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
aliyunsecbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
openindexspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
openindexdeepspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
imagesiftbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
meta-externalagent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
aliyunsecbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
meta-externalagent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
metaai
meta-ai
meta ai
meta.ai
meta-ai
metaai
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
Comments