Log In Sign Up

nzreport.com
robots.txt

Robots Exclusion Standard data for nzreport.com

Resource Scan

Scan Details

Site Domain nzreport.com
Base Domain nzreport.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan5/9/2025, 11:07:08 AM
Next Scan 6/8/2025, 11:07:08 AM

Last Successful Scan

Scanned4/10/2025, 11:01:47 AM
URL https://nzreport.com/robots.txt
Domain IPs 104.21.55.253, 172.67.174.213, 2606:4700:3031::ac43:aed5, 2606:4700:3037::6815:37fd
Response IP 172.67.174.213
Found Yes
Hash 93f116cfd408a578a51cb18d3ca9e7c6bc77b8eca90aeb15deead2e051795cc2
SimHash 601a915187e3

Groups

*

Rule Path
Disallow /articles/news/

psbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

twitterbot

Rule Path
Allow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

wget

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow
Back to top

Comments

  • Direct the most annoying crawlers not to index
  • allow social media
  • slow down the high-download crawlers
  • undesirable site scrapers and bots
  • allow useful (search engine) bots <3
Back to top