indiehackers.com
robots.txt

Robots Exclusion Standard data for indiehackers.com

Resource Scan

Scan Details

Site Domain indiehackers.com
Base Domain indiehackers.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-05T19:40:05+00:00
Next Scan 2026-01-04T19:40:05+00:00

Last Successful Scan

Scanned2024-12-17T19:30:47+00:00
URL https://indiehackers.com/robots.txt
Redirect https://www.indiehackers.com/robots.txt
Redirect Domain www.indiehackers.com
Redirect Base indiehackers.com
Domain IPs 216.24.57.252, 216.24.57.4
Redirect IPs 216.24.57.252, 216.24.57.4
Response IP 216.24.57.252
Found Yes
Hash 050061e69200ef980dd40bc84d5bcacbff1ae003b368689bc384c75bf1d0becd
SimHash 280c8820a713

Groups

*

Rule Path
Disallow

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://storage.googleapis.com/indie-hackers.appspot.com/sitemaps/ih-sitemap-index.xml

Comments

  • http://www.robotstxt.org