Log In Sign Up

astoriastorm.org
robots.txt

Robots Exclusion Standard data for astoriastorm.org

Resource Scan

Scan Details

Site Domain astoriastorm.org
Base Domain astoriastorm.org
Scan Status Ok
Last Scan2025-10-18T19:13:33+00:00
Next Scan 2025-10-25T19:13:33+00:00

Last Scan

Scanned2025-10-18T19:13:33+00:00
URL http://astoriastorm.org/robots.txt
Domain IPs 66.36.101.130
Response IP 66.36.101.130
Found Yes
Hash 3f74cda6ca445dd9f981dc3ccff04b2d2808adafcbe437a27e8ac8eb53f8c94a
SimHash 7d30dc50e052

Groups

blexbot

Rule Path
Disallow /

bonecho

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

crawler

Rule Path
Disallow /

extractor

Rule Path
Disallow /

grabber

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

nessus

Rule Path
Disallow /

panscient

Rule Path
Disallow /

parser

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

spider

Rule Path
Disallow /

teoma

Rule Path
Disallow /

yandex

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Back to top

Other Records

Field Value
sitemap https://www.hometeamsonline.com/sitemap.xml
Back to top

Comments

  • Block known bad bots (based on your dirtyBots list)
  • Allow all good bots, but restrict admin
  • Sitemap location
Back to top