Log In Sign Up

atvrider.com
robots.txt

Robots Exclusion Standard data for atvrider.com

Resource Scan

Scan Details

Site Domain atvrider.com
Base Domain atvrider.com
Scan Status Ok
Last Scan2024-11-11T01:14:59+00:00
Next Scan 2024-11-18T01:14:59+00:00

Last Scan

Scanned2024-11-11T01:14:59+00:00
URL https://atvrider.com/robots.txt
Redirect https://www.atvrider.com:443/robots.txt
Redirect Domain www.atvrider.com
Redirect Base atvrider.com
Domain IPs 15.197.174.213, 3.33.166.34
Redirect IPs 125.56.219.16, 125.56.219.27, 2600:1413:b000:13::b857:c18e, 2600:1413:b000:13::b857:c196
Response IP 23.52.171.161
Found Yes
Hash 0762a335ece945a7c7723b61443c56300b2e49414f77233d164a968ecbd97786
SimHash 08009d1ac7e9

Groups

*

Rule Path
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.gif
Allow /*.png
Allow /sites/atvrider.com/
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /sites/
Disallow /files/
Disallow /file/
Disallow /themes/
Disallow /rss/all
Disallow /comment/reply
Disallow /statistics
Disallow /customerror/404
Disallow /forward
Disallow /users
Disallow /user
Disallow /find
Disallow /privacy-policy
Disallow /terms-service
Disallow /contact-us
Disallow /abuse
Disallow /newsletter

Other Records

Field Value
crawl-delay 10

twitterbot

Rule Path
Allow *

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

asterias

Rule Path
Disallow /

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.atvrider.com/arcio/sitemap/
Back to top

Comments

  • The use of robots or other automated means to access any Bonnier Corporation website
  • without the express permission of Bonnier Corporation is strictly prohibited.
  • Bonnier Corporation may permit automated access to
  • access certain pages but solely for the limited purpose of
  • including content in publicly available search engines. Any other
  • use of robots or failure to obey the robots exclusion standards set
  • forth at http://www.robotstxt.org/wc/exclusion.html is strictly
  • prohibited.
  • Disallow the following spiders
Back to top