*
| Rule |
Path |
| Allow |
/ |
| Disallow |
/login.xhtml |
| Disallow |
/choose-agency.xhtml |
| Disallow |
/admin |
| Disallow |
/brochurePDF |
| Disallow |
/voucher/pdf |
| Disallow |
/itemvoucher/pdf |
| Disallow |
/resources |
| Disallow |
/resources-internal |
adsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
amazonbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
5 |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
gptbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
barkrowler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
claudebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
oai-searchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
chatgpt-user
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
firecrawlagent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
andibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
perplexitybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
3 |
imagesift
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
petalbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
geedoproductsearch
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
keybot translation-search-machine
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
heritrix
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
2 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
meta-externalagent
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
seekportbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
checkmarknetwork
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
1 |
crescent internet toolpak http ole control v.1.0
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
repomonkey bait & tackle/v1.01
vci webviewer vci webviewer win32
zeus 32297 webster pro v2.9 win32