housing.sfgov.org
robots.txt

Robots Exclusion Standard data for housing.sfgov.org

Resource Scan

Scan Details

Site Domain housing.sfgov.org
Base Domain sfgov.org
Scan Status Ok
Last Scan2025-12-09T12:53:55+00:00
Next Scan 2026-01-08T12:53:55+00:00

Last Scan

Scanned2025-12-09T12:53:55+00:00
URL https://housing.sfgov.org/robots.txt
Domain IPs 13.248.144.105, 13.248.241.119, 3.33.148.61, 35.71.131.46
Response IP 3.33.148.61
Found Yes
Hash b20b536e9767dbeb3feffc98e83ec155fc31a7a2cf6b0eb0d7a59149b856565b
SimHash 2c189551edf1

Groups

*
twitterbot

Rule Path
Allow /images

facebookexternalhit

Rule Path
Allow /images

Other Records

Field Value
sitemap https://housing.sfgov.org/sitemap.xml

Comments

  • Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.