Log In Sign Up

finda.co.nz
robots.txt

Robots Exclusion Standard data for finda.co.nz

Resource Scan

Scan Details

Site Domain finda.co.nz
Base Domain finda.co.nz
Scan Status Ok
Last Scan2024-11-15T01:26:08+00:00
Next Scan 2024-11-22T01:26:08+00:00

Last Scan

Scanned2024-11-15T01:26:08+00:00
URL http://finda.co.nz/robots.txt
Redirect http://www.finda.co.nz/robots.txt
Redirect Domain www.finda.co.nz
Redirect Base finda.co.nz
Domain IPs 151.138.150.91
Redirect IPs 151.138.150.91
Response IP 151.138.150.91
Found Yes
Hash 16cbf944f2524af75e1741c98d783f8f2d6d928580ac5a7f62cd3f272fd55002
SimHash 0a6ca5410fc3

Groups

httptool*

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

bordermanager*

Rule Path
Disallow /

maxthon

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

bilbo/2.3b-unix

Rule Path
Disallow /

fast-webcrawler/3.6/firstpage

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

java*

Rule Path
Disallow /

offline explorer/1.9

Rule Path
Disallow /

missigua locator 1.9

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

*

Rule Path
Disallow /clickheat/
Disallow /approve/
Disallow /business/listing/*/claim/
Disallow /afro/
Back to top

Other Records

Field Value
sitemap https://www.finda.co.nz/sitemap_index.xml
Back to top

Comments

  • robots.txt for https://www.finda.co.nz
  • Site scrapers that are completely disallowed
Back to top

Warnings

  • 4 invalid lines.
Back to top