Log In Sign Up

finda.co.nz
robots.txt

Robots Exclusion Standard data for finda.co.nz

Resource Scan

Scan Details

Site Domain finda.co.nz
Base Domain finda.co.nz
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-06-20T20:51:50+00:00
Next Scan 2024-06-27T20:51:50+00:00

Last Successful Scan

Scanned2024-06-12T20:51:38+00:00
URL http://finda.co.nz/robots.txt
Redirect http://www.finda.co.nz/robots.txt
Redirect Domain www.finda.co.nz
Redirect Base finda.co.nz
Domain IPs 151.138.150.91
Redirect IPs 151.138.150.91
Response IP 151.138.150.91
Found Yes
Hash 16cbf944f2524af75e1741c98d783f8f2d6d928580ac5a7f62cd3f272fd55002
SimHash 0a6ca5410fc3

Groups

httptool*

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

bordermanager*

Rule Path
Disallow /

maxthon

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

bilbo/2.3b-unix

Rule Path
Disallow /

fast-webcrawler/3.6/firstpage

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

java*

Rule Path
Disallow /

offline explorer/1.9

Rule Path
Disallow /

missigua locator 1.9

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

*

Rule Path
Disallow /clickheat/
Disallow /approve/
Disallow /business/listing/*/claim/
Disallow /afro/
Back to top

Other Records

Field Value
sitemap https://www.finda.co.nz/sitemap_index.xml
Back to top

Comments

  • robots.txt for https://www.finda.co.nz
  • Site scrapers that are completely disallowed
Back to top

Warnings

  • 4 invalid lines.
Back to top