orc.govt.nz
robots.txt

Robots Exclusion Standard data for orc.govt.nz

Resource Scan

Scan Details

Site Domain orc.govt.nz
Base Domain orc.govt.nz
Scan Status Ok
Last Scan2024-11-03T00:27:39+00:00
Next Scan 2024-11-17T00:27:39+00:00

Last Scan

Scanned2024-11-03T00:27:39+00:00
URL https://orc.govt.nz/robots.txt
Redirect https://www.orc.govt.nz/robots.txt
Redirect Domain www.orc.govt.nz
Redirect Base orc.govt.nz
Domain IPs 20.211.64.18
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash 3fdad17ab101c4dc9b9ca0dab573d7db6dca375c8dd044416ab300163314cc54
SimHash 00589842ee01

Groups

adbeat_bot
ahrefsbot
aitcsrobot
alexibot
blexbot
cliqzbot
dotbot
exabot
expo9
huaweisymantecspider
influencebot
ltx71 - (http://ltx71.com/)
maxpointcrawler
mj12bot
offline explorer
rogerbot
semrushbot
semrushbot-sa
sitesnagger
surveybot
teleportpro
webcopier
webreaper
webstripper
webzip
xaldon_webspider
xenu’s
xenu’s link sleuth 1.1c

Rule Path
Disallow /

*

Rule Path
Disallow /App_Plugins/
Disallow /App_Code/
Disallow /App_Data/
Disallow /bin/
Disallow /config/
Disallow /umbraco/
Disallow /Views/
Disallow /uSync/

Other Records

Field Value
sitemap https://orc.govt.nz/sitemap/

Comments

  • Exclude some crawlers