amway.ua
robots.txt

Robots Exclusion Standard data for amway.ua

Resource Scan

Scan Details

Site Domain amway.ua
Base Domain amway.ua
Scan Status Ok
Last Scan2024-09-12T14:38:34+00:00
Next Scan 2024-09-26T14:38:34+00:00

Last Scan

Scanned2024-09-12T14:38:34+00:00
URL https://amway.ua/robots.txt
Domain IPs 18.161.111.129, 18.161.111.26, 18.161.111.40, 18.161.111.69
Response IP 18.161.111.40
Found Yes
Hash 72d3e3ebead469ad08feb31f4783dd5db179f7778d6455ab5d577fb13e06a582
SimHash bcd75fbcccf8

Groups

*

Rule Path
Disallow */sso/
Disallow */singlesignon/
Disallow /?
Disallow /*icid%3D
Disallow /*cid%3D
Disallow /*aboSponsorCode%3D
Disallow /*crossSellStrategy%3D
Disallow /*refProductCode%3D
Disallow /*forceShowPLP%3D
Disallow /*product_labels%3D
Disallow /404
Disallow /ru/404
Disallow /cart
Disallow /ru/cart
Disallow /*brand%3D
Disallow /*priceValue%3D
Disallow /*pointValue%3D
Disallow /*paymentModeCode%3D
Disallow /user/
Disallow /users/
Disallow /page_000
Disallow /*%20
Disallow /sovety__
Disallow /sovety/
Disallow /ru/calendar/
Disallow /calendar/
Disallow */user/
Disallow */users/
Disallow /*code%3Den
Disallow /*escaped_fragment
Disallow /*flow%3D
Disallow */*?app=
Disallow /login/register1
Disallow /login/aboregister1
Disallow /login/aboregister2
Disallow /registration-selection
Disallow */p/
Disallow /bestsellers
Disallow /contact-us
Disallow /login/register3

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.amway.ua/sitemap.xml

Comments

  • For all robots
  • Block access to specific groups of pages
  • Insignificant parameters
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block AdsBot-Google as it's making Fake registrations

Warnings

  • `clean-param` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.