paintthetowngreen.co.uk
robots.txt

Robots Exclusion Standard data for paintthetowngreen.co.uk

Resource Scan

Scan Details

Site Domain paintthetowngreen.co.uk
Base Domain paintthetowngreen.co.uk
Scan Status Ok
Last Scan2025-04-05T14:08:59+00:00
Next Scan 2025-05-05T14:08:59+00:00

Last Scan

Scanned2025-04-05T14:08:59+00:00
URL https://paintthetowngreen.co.uk/robots.txt
Domain IPs 104.21.60.215, 172.67.201.203, 2606:4700:3033::6815:3cd7, 2606:4700:3034::ac43:c9cb
Response IP 172.67.201.203
Found Yes
Hash 18cc018a51dfb821f15b9beaba594f21bc2a7a16dddf2bebdbf17eedf6b92620
SimHash 044099036133

Groups

*

Rule Path
Allow /
Disallow /admin
Disallow /dev

Other Records

Field Value
crawl-delay 5

Comments

  • Robots.txt
  • Enables robots.txt rules for all crawlers
  • Rate limit Yahoo!, Bing and Yandex (Google ignores this)
  • SilverStripe sitemap:
  • Sitemap: https://sitedomain.com/sitemap.xml