Log In Sign Up

sthelens.com
robots.txt

Robots Exclusion Standard data for sthelens.com

Resource Scan

Scan Details

Site Domain sthelens.com
Base Domain sthelens.com
Scan Status Ok
Last Scan2025-10-04T21:30:59+00:00
Next Scan 2025-10-11T21:30:59+00:00

Last Scan

Scanned2025-10-04T21:30:59+00:00
URL https://sthelens.com/robots.txt
Domain IPs 2a02:4780:38:c58a:d26a:e0ba:a254:df19, 2a02:4780:39:800a:189c:cec:3b26:4e2b, 77.37.115.121, 93.127.187.217
Response IP 77.37.115.139
Found Yes
Hash 88519c1699b70786a13ec6777781b9a49cd1a44ae86d3f2d8afe9d8d7d13a21c
SimHash b21488209e02

Groups

claudebot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

nbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

expanse

Rule Path
Disallow /

l9explore

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

custom-asynchttpclient

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

siteuptime

Rule Path
Disallow /

genomecrawlerd

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /
Back to top