au.olight.com
robots.txt

Robots Exclusion Standard data for au.olight.com

Resource Scan

Scan Details

Site Domain au.olight.com
Base Domain olight.com
Scan Status Ok
Last Scan2025-09-12T05:13:52+00:00
Next Scan 2025-10-12T05:13:52+00:00

Last Scan

Scanned2025-09-12T05:13:52+00:00
URL https://au.olight.com/robots.txt
Domain IPs 13.35.37.103, 13.35.37.82, 13.35.37.89, 13.35.37.97
Response IP 13.35.37.97
Found Yes
Hash 52920b1bb95c1a6b546216dc6f51e263bebd36dd28c9ef9efa9d21baff609a99
SimHash 49dd5c360211

Groups

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

googlebot|bingbot|facebot

Rule Path
Allow /sitemap_*.xml

Other Records

Field Value
crawl-delay 20

facebookexternalhit

Rule Path
Allow /
Disallow /dynamic/
Disallow /search/
Disallow /admin/

Other Records

Field Value
crawl-delay 10

meta-externalfetcher

Rule Path
Allow /
Disallow /admin/
Disallow /search/
Disallow /dynamic/

Other Records

Field Value
crawl-delay 10

meta-externalagent

Rule Path
Allow /
Disallow /admin/
Disallow /search/
Disallow /dynamic/

Other Records

Field Value
crawl-delay 10

adsbot-google

Rule Path
Allow /sitemap_*.xml
Disallow /compare
Disallow /user
Disallow /login
Disallow /forgetPwd
Disallow /setting
Disallow /address
Disallow /error
Disallow /pay
Disallow /cart
Disallow /register
Disallow /checkout
Disallow /checkout?params=*
Disallow /order
Disallow /order?status=*
Disallow /myMessage
Disallow /mySales
Disallow /myStory
Disallow /myOCoin
Disallow /myBonus
Disallow /myCoupon
Disallow /affectionCard
Disallow /couponCenter
Disallow /earnRewards
Disallow /wishList
Disallow /member/pointsDetails
Disallow /search
Disallow /catalogsearch*
Disallow /search?keyword=*
Disallow /search?couponId=*
Disallow /%20
Disallow /evaluation
Disallow /landing-page
Disallow /exclusive_benefit
Disallow /back-program
Disallow /repeat_customer_program
Disallow /*-copy
Disallow /*-copy-*
Disallow /*-coin
Disallow /*-ocoin
Disallow /*-diamond-*
Disallow /recall.html?id=*
Disallow /recall/agreement
Disallow /activities.html?id=*
Disallow /s/*
Disallow /suggestions
Disallow /*skuId%3D*
Disallow /*productId%3D*
Disallow /*blogLabelId%3D*
Disallow /blog/detail.html?id=*
Disallow /article.html?id=*
Disallow /*.pdf$

Comments

  • Block Ahrefs crawlers
  • Google/Bing/Facebot 特殊配置
  • 限制 FacebookExternalHit 爬虫
  • 限制 meta-externalfetcher
  • 限制 meta-externalagent