academicbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
amazonbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
applebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
brightbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
chatgpt-user
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
claudebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dataforseobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gensparkbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gptbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
googlebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mojeekbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
oai-searchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
perplexitybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
petalbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
poesearchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
qwantbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semanticscholarbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
seocherrybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
siteauditbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
slackbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
yandexrenderresourcesbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
academicbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
amazonbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
applebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
bitlybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
gptbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
meta-externalagent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
node-fetch
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
petalbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
searchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
*
| Rule |
Path |
| Disallow |
|
| Disallow |
/cgi-bin/ |
Comments