Log In Sign Up

utvdriver.com
robots.txt

Robots Exclusion Standard data for utvdriver.com

Resource Scan

Scan Details

Site Domain utvdriver.com
Base Domain utvdriver.com
Scan Status Ok
Last Scan2024-06-27T22:17:17+00:00
Next Scan 2024-07-04T22:17:17+00:00

Last Scan

Scanned2024-06-27T22:17:17+00:00
URL https://utvdriver.com/robots.txt
Redirect https://www.utvdriver.com:443/robots.txt
Redirect Domain www.utvdriver.com
Redirect Base utvdriver.com
Domain IPs 15.197.174.213, 3.33.166.34
Redirect IPs 125.56.219.58, 125.56.219.64, 2600:1413:b000:14::b857:c155, 2600:1413:b000:14::b857:c158
Response IP 125.56.219.58
Found Yes
Hash efd13b500134b03338a53a449efef044175ebe3845d6d56d6346066a3302882b
SimHash 20009d1ac7e5

Groups

*

Rule Path
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.gif
Allow /*.png
Allow /sites/utvdriver.com/
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /sites/
Disallow /files/
Disallow /file/
Disallow /themes/
Disallow /rss/all
Disallow /comment/reply
Disallow /statistics
Disallow /customerror/404
Disallow /forward
Disallow /users
Disallow /user
Disallow /find
Disallow /privacy-policy
Disallow /terms-service
Disallow /contact-us
Disallow /abuse
Disallow /newsletter

Other Records

Field Value
crawl-delay 10

twitterbot

Rule Path
Allow *

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.utvdriver.com/arcio/sitemap/
Back to top

Comments

  • The use of robots or other automated means to access any Bonnier Corporation website
  • without the express permission of Bonnier Corporation is strictly prohibited.
  • Bonnier Corporation may permit automated access to
  • access certain pages but solely for the limited purpose of
  • including content in publicly available search engines. Any other
  • use of robots or failure to obey the robots exclusion standards set
  • forth at http://www.robotstxt.org/wc/exclusion.html is strictly
  • prohibited.
  • Disallow the following spiders
Back to top