Log In Sign Up

webhostingmagic.com
robots.txt

Robots Exclusion Standard data for webhostingmagic.com

Resource Scan

Scan Details

Site Domain webhostingmagic.com
Base Domain webhostingmagic.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-06T05:43:47+00:00
Next Scan 2026-01-05T05:43:47+00:00

Last Successful Scan

Scanned2025-08-30T21:20:55+00:00
URL https://webhostingmagic.com/robots.txt
Domain IPs 104.197.200.143
Response IP 104.197.200.143
Found Yes
Hash e60c14756112ae8d7d3d7b1b738d0e55cf913b85dda0baa5c0b999df62aadf8b
SimHash 525088c2c173

Groups

*

Rule Path
Allow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googlebot-extended

Rule Path
Disallow /

bard-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

bytespider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.webhostingmagic.com/sitemap.xml
sitemap https://www.webhostingmagic.com/sitemap_index.xml
sitemap https://www.webhostingmagic.com/sitemap.txt
Back to top

Comments

  • Default rule for all bots
  • Block AI and LLM-related crawlers
  • Sitemap declarations
Back to top