streetru.co.uk
robots.txt

Robots Exclusion Standard data for streetru.co.uk

Resource Scan

Scan Details

Site Domain streetru.co.uk
Base Domain streetru.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-21T10:47:59+00:00
Next Scan 2024-07-05T10:47:59+00:00

Last Successful Scan

Scanned2024-06-06T10:47:08+00:00
URL https://streetru.co.uk/robots.txt
Redirect https://www.streetru.co.uk/robots.txt
Redirect Domain www.streetru.co.uk
Redirect Base streetru.co.uk
Domain IPs 13.33.21.102, 13.33.21.13, 13.33.21.44, 13.33.21.77
Redirect IPs 13.226.2.30, 13.226.2.58, 13.226.2.75, 13.226.2.76
Response IP 108.157.60.74
Found Yes
Hash 17b3fd0c053d010b1048f9ebed306bab270e11496c21cdad09f2f45167fd4c48
SimHash bb548f400f51

Groups

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.streetru.co.uk/sitemap.xml

Comments

  • Robots.txt file
  • All robots will spider the domain