house-improvements.com
robots.txt

Robots Exclusion Standard data for house-improvements.com

Resource Scan

Scan Details

Site Domain house-improvements.com
Base Domain house-improvements.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-18T15:21:18+00:00
Next Scan 2025-12-17T15:21:18+00:00

Last Successful Scan

Scanned2025-08-20T12:28:19+00:00
URL https://house-improvements.com/robots.txt
Redirect https://www.house-improvements.com/robots.txt
Redirect Domain www.house-improvements.com
Redirect Base house-improvements.com
Domain IPs 104.21.42.173, 172.67.206.208, 2606:4700:3030::6815:2aad, 2606:4700:3033::ac43:ced0
Redirect IPs 104.21.42.173, 172.67.206.208, 2606:4700:3030::6815:2aad, 2606:4700:3033::ac43:ced0
Response IP 172.67.206.208
Found Yes
Hash fc92c280eed1ba88da6f2f57679c15163c4bf7edf0c8b95839720a86b12bd33f
SimHash 4c56dbd070d6

Groups

*

Rule Path
Disallow /

bingbot
bingpreview
slurp
applebot
googlebot
googlebot-image
mediapartners-google
archive.org_bot
baiduspider
duckduckbot

Rule Path
Disallow

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2.0

Comments

  • House last updated 2020-01-30
  • Block everyone first
  • Except allowed bots
  • 2020-01-31 slow down crawlers that don't send much traffic