Log In Sign Up

makeupalley.com
robots.txt

Robots Exclusion Standard data for makeupalley.com

Resource Scan

Scan Details

Site Domain makeupalley.com
Base Domain makeupalley.com
Scan Status Ok
Last Scan2024-05-29T09:17:26+00:00
Next Scan 2024-06-05T09:17:26+00:00

Last Scan

Scanned2024-05-29T09:17:26+00:00
URL https://makeupalley.com/robots.txt
Redirect https://www.makeupalley.com/robots.txt
Redirect Domain www.makeupalley.com
Redirect Base makeupalley.com
Domain IPs 99.84.66.20, 99.84.66.30, 99.84.66.35, 99.84.66.63
Redirect IPs 65.8.161.128, 65.8.161.39, 65.8.161.83, 65.8.161.88
Response IP 108.157.60.104
Found Yes
Hash da5c09a7cc2ea4f443248bf09e3dd40fec1f81ae248811351cd3720d5f399deb
SimHash 4824c291c012

Groups

googlebot

Rule Path
Disallow /links/out
Disallow /account/
Disallow /mail/
Disallow /product/qu
Disallow /product/x_qu
Disallow /product/review
Disallow /product/__reviews
Disallow /product/__brand
Disallow /__home
Disallow /product/addreview
Disallow /product/addproduct
Disallow /*p~*
Disallow /m_*

mediapartners-google

Rule Path
Disallow /account/act
Disallow /product/qu
Disallow /account/email
Disallow /account/remove

slurp

Rule Path
Disallow /links/out
Disallow /product/myreviews
Disallow /account/
Disallow /mail/
Disallow /product/qu
Disallow /product/review
Disallow /product/__reviews
Disallow /product/__brand
Disallow /__home
Disallow /product/addreview
Disallow /product/addproduct

Other Records

Field Value
crawl-delay 10

daumoa

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot/2.0; +http://go.mail.ru/help/robots

Rule Path
Disallow /

bingbot

Rule Path
Disallow /links/out
Disallow /product/myreviews
Disallow /account/
Disallow /mail/
Disallow /product/qu
Disallow /product/x_qu
Disallow /product/review
Disallow /product/__reviews
Disallow /product/__brand
Disallow /__home
Disallow /product/sreview/
Disallow /product/addreview
Disallow /product/addproduct
Disallow /*p~*
Disallow /m_*
Disallow /product/searching

msnbot

Rule Path
Disallow /links/out
Disallow /product/myreviews
Disallow /account/
Disallow /mail/
Disallow /product/qu
Disallow /product/x_qu
Disallow /product/review
Disallow /product/__reviews
Disallow /product/__brand
Disallow /__home
Disallow /product/sreview/
Disallow /product/addreview
Disallow /product/addproduct
Disallow /*p~*
Disallow /m_*
Disallow /product/searching

*

Rule Path
Disallow /links/out
Disallow /product/myreviews
Disallow /account/
Disallow /mail/
Disallow /product/qu
Disallow /product/x_qu
Disallow /product/review
Disallow /product/__reviews
Disallow /product/__brand
Disallow /__home
Disallow /product/sreview/
Disallow /product/addreview
Disallow /product/addproduct
Disallow /*p~*
Disallow /m_*
Disallow /__brand/

etaospider

Rule Path
Disallow /

mozilla/5.0 (compatible; etaospider/1.0; http://open.etao.com/dev/etaospider)

Rule Path
Disallow /

mozilla/5.0+(compatible;+etaospider/1.0;+http://open.etao.com/dev/etaospider)

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sogou-test-spider/4.0 (compatible; msie 5.5; windows 98)

Rule Path
Disallow /

yandex

Rule Path
Disallow /

proximic

Rule Path
Disallow /
Back to top

Comments

  • Mail.RU_Bot/2.0 Russia
  • Mail.RU Russia
  • Mail.Ru Russia
  • Block Yandex
  • proximic.com/info/spider.php
Back to top