Log In Sign Up

dogfriendlycottages.co.uk
robots.txt

Robots Exclusion Standard data for dogfriendlycottages.co.uk

Resource Scan

Scan Details

Site Domain dogfriendlycottages.co.uk
Base Domain dogfriendlycottages.co.uk
Scan Status Ok
Last Scan2024-10-18T09:14:05+00:00
Next Scan 2024-11-17T09:14:05+00:00

Last Scan

Scanned2024-10-18T09:14:05+00:00
URL https://dogfriendlycottages.co.uk/robots.txt
Redirect https://www.dogfriendlycottages.co.uk/robots.txt
Redirect Domain www.dogfriendlycottages.co.uk
Redirect Base dogfriendlycottages.co.uk
Domain IPs 2600:9000:21b7:1000:11:9460:3e40:93a1, 2600:9000:21b7:1e00:11:9460:3e40:93a1, 2600:9000:21b7:2400:11:9460:3e40:93a1, 2600:9000:21b7:3a00:11:9460:3e40:93a1, 2600:9000:21b7:5400:11:9460:3e40:93a1, 2600:9000:21b7:8a00:11:9460:3e40:93a1, 2600:9000:21b7:b200:11:9460:3e40:93a1, 2600:9000:21b7:e400:11:9460:3e40:93a1, 3.166.96.13, 3.166.96.39, 3.166.96.58, 3.166.96.88
Redirect IPs 18.155.68.34, 18.155.68.79, 18.155.68.85, 18.155.68.97, 2600:9000:23d2:2200:11:9460:3e40:93a1, 2600:9000:23d2:2a00:11:9460:3e40:93a1, 2600:9000:23d2:4600:11:9460:3e40:93a1, 2600:9000:23d2:6200:11:9460:3e40:93a1, 2600:9000:23d2:9c00:11:9460:3e40:93a1, 2600:9000:23d2:aa00:11:9460:3e40:93a1, 2600:9000:23d2:de00:11:9460:3e40:93a1, 2600:9000:23d2:fe00:11:9460:3e40:93a1
Response IP 18.155.68.97
Found Yes
Hash eedc1205a5b1b3ccd34b130db2ee9a3c2f789dcf2f8fc720954186537d249ca4
SimHash a81b2b02a47f

Groups

yandex

Rule Path
Disallow /api/
Disallow /searches/
Disallow /favourites_lists/
Allow /searches/gbs-search/

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /api/
Disallow /searches/
Disallow /favourites_lists/
Disallow *utm_source%3D*
Disallow *check_in_on%3D*
Allow /searches/gbs-search/
Allow /searches/*.json
Back to top

Other Records

Field Value
sitemap https://www.dogfriendlycottages.co.uk/sitemaps/index.xml
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
Back to top