fairfaxmedia.co.nz
robots.txt

Robots Exclusion Standard data for fairfaxmedia.co.nz

Resource Scan

Scan Details

Site Domain fairfaxmedia.co.nz
Base Domain fairfaxmedia.co.nz
Scan Status Ok
Last Scan2024-11-11T22:35:44+00:00
Next Scan 2024-11-18T22:35:44+00:00

Last Scan

Scanned2024-11-11T22:35:44+00:00
URL https://fairfaxmedia.co.nz/robots.txt
Redirect https://www.stuff.co.nz/robots.txt
Redirect Domain www.stuff.co.nz
Redirect Base stuff.co.nz
Domain IPs 13.35.210.23, 13.35.210.29, 13.35.210.60, 13.35.210.71
Redirect IPs 151.101.130.227, 151.101.194.227, 151.101.2.227, 151.101.66.227, 2a04:4e42:200::739, 2a04:4e42:400::739, 2a04:4e42:600::739, 2a04:4e42::739
Response IP 199.232.46.227
Found Yes
Hash de902921e90bd9b49dafb4924e60fa507b9f64a2f30718e23c63be9f48386b68
SimHash 107899d1a5f0

Groups

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /essentialmums/
Disallow /email_a_friend/
Disallow /entertainment/bravo

*

Rule Path
Disallow /new-relic/stuff-new-relic-prod.js
Disallow /static/stuff-adtech-sdk/ads-sdk/master/main.js
Disallow /static/stuff-adtech-sdk/ads-sdk/master/prebid.js
Disallow /static/stuff-adtech-sdk/ads-sdk/master/video-ads.main.js
Disallow /static/stuff-web/script/iframeResizer.contentWindow.min.js
Disallow /static/stuff-web/script/iframeResizer.min.js
Disallow /spade-widget/*

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://stuff.co.nz/sitemap.xml

Comments

  • robots for https://stuff.co.nz
  • allowing grapeshot to access to content
  • Disallowed paths
  • Block specific JS files for all user agents
  • Site Scrapers and bots that are not desirable: