Log In Sign Up

purplevrs.com
robots.txt

Robots Exclusion Standard data for purplevrs.com

Resource Scan

Scan Details

Site Domain purplevrs.com
Base Domain purplevrs.com
Scan Status Ok
Last Scan2024-09-21T07:48:32+00:00
Next Scan 2024-10-21T07:48:32+00:00

Last Scan

Scanned2024-09-21T07:48:32+00:00
URL https://purplevrs.com/robots.txt
Domain IPs 208.17.91.50
Response IP 208.17.91.50
Found Yes
Hash 8f31a777bcc22ab8b88da60cf6f13b8e49fc82f394a91d55d25a2907ecce375c
SimHash bb5c07468039

Groups

sosospider

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

yandex

Rule Path
Disallow /

Other Records

Field Value Comment
crawl-delay 500 specifies a 500 seconds timeout

sogou

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

googlebot

Rule Path
Disallow
Allow /

mediapartners-google

Rule Path
Disallow
Allow /

proximic

Rule Path
Disallow /

*

Rule Path
Disallow /resizer.aspx
Disallow /Services/
Disallow /css/HttpCombiner.ashx
Disallow /*.axd
Disallow /umbraco/

Other Records

Field Value
crawl-delay 600
Back to top

Other Records

Field Value
sitemap /sitemap-dynamic.aspx
Back to top

Comments

  • Deny Soso spider in the site <http://help.soso.com/webspider.htm>
  • Deny EC2LinkFinder in the site
  • deny 80legs.com webcrawler
  • deny http://www.metadatalabs.com/mlbot
  • User-agent: MLBot
  • Disallow: /
  • deny yandex http://yandex.com/bots
  • deny Sogou spider http://www.sogou.com/docs/help/webmasters.htm#07
  • <http://www.youdao.com/help/webmaster/spider/>
  • <http://help.naver.com/robots/>
  • <http://help.goo.ne.jp/door/crawler.html>
  • Ils font payer les donnĂ©es d'autrui.
  • <http://spinn3r.com/robot>
  • google case
  • User-agent: Googlebot
  • Disallow: /php3
  • Allow: /
  • proximic used by amazon EC2 cloud, we already blacklisted some of their IPs for abuse
  • general case
  • specifies a 600 seconds timeout
Back to top

Warnings

  • 2 invalid lines.
Back to top