medialens.org
robots.txt

Robots Exclusion Standard data for medialens.org

Resource Scan

Scan Details

Site Domain medialens.org
Base Domain medialens.org
Scan Status Ok
Last Scan2026-02-06T20:32:37+00:00
Next Scan 2026-03-08T20:32:37+00:00

Last Scan

Scanned2026-02-06T20:32:37+00:00
URL https://medialens.org/robots.txt
Domain IPs 104.21.87.105, 172.67.169.95, 2606:4700:3030::ac43:a95f, 2606:4700:3037::6815:5769
Response IP 172.67.169.95
Found Yes
Hash 51ac6dbe4bf5aef0e34360785d3219bba3361f7076e4aeaa3d79a4e244d1861a
SimHash 0100da02cfb0

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

semrushbot

Rule Path
Disallow /

Comments

  • Default
  • Block SemRush
  • No search engines
  • User-agent: *
  • Disallow: /
  • But allow Twitter
  • User-agent: Twitterbot
  • Disallow: