yesauto.co.uk
robots.txt

Robots Exclusion Standard data for yesauto.co.uk

Resource Scan

Scan Details

Site Domain yesauto.co.uk
Base Domain yesauto.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-08T21:47:06+00:00
Next Scan 2024-07-07T21:47:06+00:00

Last Successful Scan

Scanned2022-06-28T00:05:37+00:00
URL https://www.yesauto.co.uk/robots.txt
Response IP 54.192.150.110
Found Yes
Hash 14e8aca93d2988415159b437f77c8db68004892d040a4027ac3b3245d89780e3
SimHash 65058139e3e0

Groups

adsbot-google

Rule Path
Disallow
Disallow /my
Disallow /login

*

Rule Path
Disallow /videos/user-stories/comments
Disallow /user-reviews/
Disallow /user/
Disallow /search
Disallow /review/publish
Disallow /review/norm
Disallow /news/tgc/
Disallow /news/topics/
Disallow /content/article/extern
Disallow /content/video/extern
Disallow /my
Disallow /login
Disallow /feedback
Disallow /download
Disallow /configurator
Disallow /compare
Disallow /car-leasing/detail/
Disallow /404
Disallow /uk/
Disallow /product/

baiduspider

Rule Path
Disallow /*
Allow /$

Other Records

Field Value
sitemap https://www.yesauto.co.uk/sitemap.xml
sitemap https://www.yesauto.co.uk/rss.xml

Comments

  • limit Baiduspider to crawl homepage, and homepage only