Log In Sign Up

capstone.hk
robots.txt

Robots Exclusion Standard data for capstone.hk

Resource Scan

Scan Details

Site Domain capstone.hk
Base Domain capstone.hk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-09-21T08:05:27+00:00
Next Scan 2024-10-05T08:05:27+00:00

Last Successful Scan

Scanned2024-08-14T07:38:15+00:00
URL http://capstone.hk/robots.txt
Redirect https://shop.capstone.hk/robots.txt
Redirect Domain shop.capstone.hk
Redirect Base capstone.hk
Domain IPs 180.92.182.137
Redirect IPs 13.214.255.32, 54.169.62.94
Response IP 54.169.62.94
Found Yes
Hash 8efe6793178529fb8112b71b1fb35a365c3aad11b2b58d9a2e91a19f2c467819
SimHash a15c9f01cdd7

Groups

mj12bot

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

*

Rule Path
Disallow /closed
Disallow /preview/
Disallow /users/
Disallow /orders
Disallow /*?*debug=*
Disallow /*?*theme_preview=*
Disallow /*?*price_range_preview=*
Disallow /*?*draft=*
Disallow /api/
Disallow /themes/
Disallow /products*?*query=*
Back to top

Other Records

Field Value
sitemap https://shop.capstone.hk/sitemap.xml
Back to top

Comments

  • robots.txt file for Shopline Merchant
  • split user-agent disallows for different bots, as not all bots may follow google's multi-user-agent standard
  • Allow crawling of all content except
Back to top