Log In Sign Up

purple.us
robots.txt

Robots Exclusion Standard data for purple.us

Resource Scan

Scan Details

Site Domain purple.us
Base Domain purple.us
Scan Status Ok
Last Scan2024-09-01T00:38:11+00:00
Next Scan 2024-10-01T00:38:11+00:00

Last Scan

Scanned2024-09-01T00:38:11+00:00
URL https://purple.us/robots.txt
Redirect https://www.purplevrs.com/robots.txt
Redirect Domain www.purplevrs.com
Redirect Base purplevrs.com
Domain IPs 208.17.91.118
Redirect IPs 208.17.91.50
Response IP 208.17.91.50
Found Yes
Hash 8f31a777bcc22ab8b88da60cf6f13b8e49fc82f394a91d55d25a2907ecce375c
SimHash bb5c07468039

Groups

sosospider

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

yandex

Rule Path
Disallow /

Other Records

Field Value Comment
crawl-delay 500 specifies a 500 seconds timeout

sogou

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

googlebot

Rule Path
Disallow
Allow /

mediapartners-google

Rule Path
Disallow
Allow /

proximic

Rule Path
Disallow /

*

Rule Path
Disallow /resizer.aspx
Disallow /Services/
Disallow /css/HttpCombiner.ashx
Disallow /*.axd
Disallow /umbraco/

Other Records

Field Value
crawl-delay 600
Back to top

Other Records

Field Value
sitemap /sitemap-dynamic.aspx
Back to top

Comments

  • Deny Soso spider in the site <http://help.soso.com/webspider.htm>
  • Deny EC2LinkFinder in the site
  • deny 80legs.com webcrawler
  • deny http://www.metadatalabs.com/mlbot
  • User-agent: MLBot
  • Disallow: /
  • deny yandex http://yandex.com/bots
  • deny Sogou spider http://www.sogou.com/docs/help/webmasters.htm#07
  • <http://www.youdao.com/help/webmaster/spider/>
  • <http://help.naver.com/robots/>
  • <http://help.goo.ne.jp/door/crawler.html>
  • Ils font payer les donnĂ©es d'autrui.
  • <http://spinn3r.com/robot>
  • google case
  • User-agent: Googlebot
  • Disallow: /php3
  • Allow: /
  • proximic used by amazon EC2 cloud, we already blacklisted some of their IPs for abuse
  • general case
  • specifies a 600 seconds timeout
Back to top

Warnings

  • 2 invalid lines.
Back to top