social.yesterweb.org
robots.txt

Robots Exclusion Standard data for social.yesterweb.org

Resource Scan

Scan Details

Site Domain social.yesterweb.org
Base Domain yesterweb.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-30T08:44:44+00:00
Next Scan 2024-12-29T08:44:44+00:00

Last Successful Scan

Scanned2024-03-11T06:30:13+00:00
URL https://social.yesterweb.org/robots.txt
Domain IPs 68.133.1.71
Response IP 68.133.1.71
Found Yes
Hash 6ef40c9c275c7394e3efbfd0bb8bcdd1c54c0bf54bb000991b9c64e823815783
SimHash aa149b85b262

Groups

*
fedicrawl/1.0
fedimapper
fediversealmanac
fedidb/0.5.0; +https://fedidb.org/crawler.html

Rule Path
Disallow /

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • User-agent: *
  • Disallow: /media_proxy/
  • Disallow: /interact/