snarfed.org
robots.txt

Robots Exclusion Standard data for snarfed.org

Resource Scan

Scan Details

Site Domain snarfed.org
Base Domain snarfed.org
Scan Status Ok
Last Scan2025-05-19T16:17:33+00:00
Next Scan 2025-06-18T16:17:33+00:00

Last Scan

Scanned2025-05-19T16:17:33+00:00
URL https://snarfed.org/robots.txt
Redirect https://snarfed.org/s/robots.txt
Domain IPs 192.0.78.180, 192.0.78.215
Response IP 192.0.78.215
Found Yes
Hash 066cda2508027cbfbd8d830a5826c4a0eccc5cf9c5f18b1fce6a4e066ed51d1f
SimHash f8dcf1019ff1

Groups

*

Rule Path
Disallow /category/
Disallow /gallery/
Disallow /page/
Disallow /search/
Disallow /tag/
Disallow /unlisted
Disallow /wp-admin/
Disallow *.txt

Comments

  • the globbing here is nonstandard; it only works with google.