murdoch-murdoch.net
robots.txt

Robots Exclusion Standard data for murdoch-murdoch.net

Resource Scan

Scan Details

Site Domain murdoch-murdoch.net
Base Domain murdoch-murdoch.net
Scan Status Ok
Last Scan2025-11-25T05:57:52+00:00
Next Scan 2025-12-25T05:57:52+00:00

Last Scan

Scanned2025-11-25T05:57:52+00:00
URL https://murdoch-murdoch.net/robots.txt
Domain IPs 104.21.23.120, 172.67.210.215, 2606:4700:3033::6815:1778, 2606:4700:3037::ac43:d2d7
Response IP 104.21.23.120
Found Yes
Hash 0f2f78c6a29804094a7572feeec46f9a6554aadedca6f53f2ed58b61476b0334
SimHash b514d93edf55

Groups

*

Rule Path
Allow /
Disallow /videos
Disallow /audio
Disallow /hls
Disallow /transcripts
Disallow /data
Disallow /js
Disallow /css
Disallow /scripts
Disallow /ath/audio
Disallow /ath/pdf
Disallow /ath/images
Disallow /html/ath

mj12bot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

Comments

  • badly-behaved bots
  • MJ12Bot was downloading audio - wtf.
  • It's greedy, but it feeds Alexa per the bot web-page, so may as well allow it.
  • User-agent: Amazonbot
  • Disallow: /