masseypress.ac.nz
robots.txt

Robots Exclusion Standard data for masseypress.ac.nz

Resource Scan

Scan Details

Site Domain masseypress.ac.nz
Base Domain masseypress.ac.nz
Scan Status Ok
Last Scan2024-05-27T14:26:32+00:00
Next Scan 2024-06-26T14:26:32+00:00

Last Scan

Scanned2024-05-27T14:26:32+00:00
URL https://masseypress.ac.nz/robots.txt
Redirect https://www.masseypress.ac.nz/robots.txt
Redirect Domain www.masseypress.ac.nz
Redirect Base masseypress.ac.nz
Domain IPs 54.206.41.86
Redirect IPs 54.206.41.86
Response IP 54.206.41.86
Found Yes
Hash a106ccbd067928c7cba7f265a7aadd8fe8334f266908ca04f4707a5a182f4b51
SimHash 005c9842ec01

Groups

adbeat_bot
ahrefsbot
aitcsrobot
alexibot
blexbot
cliqzbot
dotbot
exabot
expo9
huaweisymantecspider
influencebot
ltx71 - (http://ltx71.com/)
maxpointcrawler
mj12bot
offline explorer
rogerbot
semrushbot
semrushbot-sa
sitesnagger
surveybot
teleportpro
webcopier
webreaper
webstripper
webzip
xaldon_webspider
xenu’s
xenu’s link sleuth 1.1c

Rule Path
Disallow /

*

Rule Path
Disallow /App_Plugins/
Disallow /App_Code/
Disallow /App_Data/
Disallow /bin/
Disallow /config/
Disallow /umbraco/
Disallow /Views/
Disallow /uSync/

Other Records

Field Value
sitemap https://masseypress.ac.nz/sitemap/

Comments

  • Exclude some crawlers