Log In Sign Up

versaweb.net
robots.txt

Robots Exclusion Standard data for versaweb.net

Resource Scan

Scan Details

Site Domain versaweb.net
Base Domain versaweb.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2025-12-25T17:07:51+00:00
Next Scan 2026-01-08T17:07:51+00:00

Last Successful Scan

Scanned2025-12-03T10:18:31+00:00
URL https://versaweb.net/robots.txt
Domain IPs 172.93.1.12, 2604:2280:dead:beef::1:12
Response IP 172.93.1.12
Found Yes
Hash 0de498ff61c584c8a638f3650358a74f0c5f88f9aa0b5f7467f980d5d63386e5
SimHash 40101315e50b

Groups

googlebot
googlebot-news
bingbot
yandexbot
duckduckbot
bravebot
baidubot
baiduspider
slurp
perplexitybot
ahrefsbot
semrushbot
serverhunter

Rule Path
Allow /

gptbot
oai-searchbot
chatgpt-user
perplexitybot
anthropic
grok
meta-externalagent
amazonbot
applebot
claudebot
bytespider

Rule Path
Allow /

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.versaweb.com/sitemap.xml
Back to top

Comments

  • Sitemap
  • Block specific directories for all crawlers
  • Allow significant search engines
  • Allow AI LLMs (broad approach due to unclear crawler names)
  • Block all other crawlers
Back to top

Warnings

  • 7 invalid lines.
Back to top