Log In Sign Up

theblackbow.com.my
robots.txt

Robots Exclusion Standard data for theblackbow.com.my

Resource Scan

Scan Details

Site Domain theblackbow.com.my
Base Domain theblackbow.com.my
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-22T15:55:10+00:00
Next Scan 2024-07-06T15:55:10+00:00

Last Successful Scan

Scanned2024-05-15T15:54:32+00:00
URL https://www.theblackbow.com.my/robots.txt
Domain IPs 13.225.4.114, 13.225.4.120, 13.225.4.125, 13.225.4.27
Response IP 13.225.4.27
Found Yes
Hash 209d64c728186a069b85f1aa8ae3453e1d09cc258283789b4c751189bed32ba1
SimHash 295c9f11cdd6

Groups

mj12bot

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

*

Rule Path
Disallow /closed
Disallow /preview/
Disallow /users/
Disallow /orders
Disallow /*?*debug=*
Disallow /*?*theme_preview=*
Disallow /*?*price_range_preview=*
Disallow /*?*draft=*
Disallow /api/
Disallow /themes/
Disallow /products*?*query=*
Back to top

Other Records

Field Value
sitemap https://www.theblackbow.com.my/sitemap.xml
Back to top

Comments

  • robots.txt file for Shopline Merchant
  • split user-agent disallows for different bots, as not all bots may follow google's multi-user-agent standard
  • Allow crawling of all content except
Back to top