Log In Sign Up

dogfriendlycottages.co.uk
robots.txt

Robots Exclusion Standard data for dogfriendlycottages.co.uk

Resource Scan

Scan Details

Site Domain dogfriendlycottages.co.uk
Base Domain dogfriendlycottages.co.uk
Scan Status Ok
Last Scan2024-05-21T07:42:00+00:00
Next Scan 2024-06-20T07:42:00+00:00

Last Scan

Scanned2024-05-21T07:42:00+00:00
URL https://dogfriendlycottages.co.uk/robots.txt
Redirect https://www.dogfriendlycottages.co.uk/robots.txt
Redirect Domain www.dogfriendlycottages.co.uk
Redirect Base dogfriendlycottages.co.uk
Domain IPs 108.157.142.106, 108.157.142.129, 108.157.142.51, 108.157.142.64, 2600:9000:21b7:2600:11:9460:3e40:93a1, 2600:9000:21b7:7400:11:9460:3e40:93a1, 2600:9000:21b7:8e00:11:9460:3e40:93a1, 2600:9000:21b7:a800:11:9460:3e40:93a1, 2600:9000:21b7:b600:11:9460:3e40:93a1, 2600:9000:21b7:c200:11:9460:3e40:93a1, 2600:9000:21b7:f200:11:9460:3e40:93a1, 2600:9000:21b7:f600:11:9460:3e40:93a1
Redirect IPs 18.155.68.34, 18.155.68.79, 18.155.68.85, 18.155.68.97, 2600:9000:23d2:1400:11:9460:3e40:93a1, 2600:9000:23d2:2400:11:9460:3e40:93a1, 2600:9000:23d2:4e00:11:9460:3e40:93a1, 2600:9000:23d2:6000:11:9460:3e40:93a1, 2600:9000:23d2:7a00:11:9460:3e40:93a1, 2600:9000:23d2:8000:11:9460:3e40:93a1, 2600:9000:23d2:9800:11:9460:3e40:93a1, 2600:9000:23d2:e00:11:9460:3e40:93a1
Response IP 18.155.68.85
Found Yes
Hash eedc1205a5b1b3ccd34b130db2ee9a3c2f789dcf2f8fc720954186537d249ca4
SimHash a81b2b02a47f

Groups

yandex

Rule Path
Disallow /api/
Disallow /searches/
Disallow /favourites_lists/
Allow /searches/gbs-search/

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /api/
Disallow /searches/
Disallow /favourites_lists/
Disallow *utm_source%3D*
Disallow *check_in_on%3D*
Allow /searches/gbs-search/
Allow /searches/*.json
Back to top

Other Records

Field Value
sitemap https://www.dogfriendlycottages.co.uk/sitemaps/index.xml
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
Back to top