Log In Sign Up

comparebear.co.nz
robots.txt

Robots Exclusion Standard data for comparebear.co.nz

Resource Scan

Scan Details

Site Domain comparebear.co.nz
Base Domain comparebear.co.nz
Scan Status Ok
Last Scan2024-05-16T21:13:12+00:00
Next Scan 2024-06-15T21:13:12+00:00

Last Scan

Scanned2024-05-16T21:13:12+00:00
URL https://comparebear.co.nz/robots.txt
Redirect https://www.comparebear.co.nz/robots.txt
Redirect Domain www.comparebear.co.nz
Redirect Base comparebear.co.nz
Domain IPs 104.21.77.5, 172.67.202.197, 2606:4700:3031::ac43:cac5, 2606:4700:3033::6815:4d05
Redirect IPs 104.21.77.5, 172.67.202.197, 2606:4700:3031::ac43:cac5, 2606:4700:3033::6815:4d05
Response IP 104.21.77.5
Found Yes
Hash a434c43a2e91af8f8243d2c72b08fbc5b482ee983612b48782d306abcfa61c1d
SimHash b28d2c87e770

Groups

*

Rule Path
Disallow /login_stakeholder
Disallow /reporting/*
Disallow /api/*
Back to top

Other Records

Field Value
sitemap https://www.comparebear.co.nz/sitemap.xml
Back to top

Comments

  • (moved from public/robots.txt)
  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top