natlib.govt.nz
robots.txt

Robots Exclusion Standard data for natlib.govt.nz

Resource Scan

Scan Details

Site Domain natlib.govt.nz
Base Domain natlib.govt.nz
Scan Status Ok
Last Scan2024-09-24T23:02:57+00:00
Next Scan 2024-10-24T23:02:57+00:00

Last Scan

Scanned2024-09-24T23:02:57+00:00
URL https://natlib.govt.nz/robots.txt
Domain IPs 45.60.14.156, 45.60.16.156
Response IP 45.60.14.156
Found Yes
Hash db80a13a4e7580f8086945fcf5c27b3711394ca7e7936eee01a60a4c2a39466b
SimHash 986c31ed9fc1

Groups

*

Rule Path
Disallow /auth
Disallow /cart_items
Disallow /shop/cart_items
Disallow /viewing_cart_items
Disallow /items/favourites
Disallow /files
Disallow /logicrouter
Disallow /primo_library
Disallow /questions
Disallow /items?*page=*
Disallow /plug/api/site_notices*
Disallow /headings
Disallow /records/*/enquiries/new

facebookexternalhit/*

Rule Path
Disallow /404
Disallow /auth

twitterbot

Rule Path
Disallow /404
Disallow /auth

cliqzbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file