Log In Sign Up

api.watsons.co.th
robots.txt

Robots Exclusion Standard data for api.watsons.co.th

Resource Scan

Scan Details

Site Domain api.watsons.co.th
Base Domain watsons.co.th
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-01T00:44:20+00:00
Next Scan 2025-12-30T00:44:20+00:00

Last Successful Scan

Scanned2021-08-01T02:33:56+00:00
URL https://api.watsons.co.th/robots.txt
Found Yes
Hash 59485efad8a0ba568348d8aeac8ab9e871129fdb593047a1e50545655f1f54f4
SimHash ec605f1cede0

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://api.watsons.com.ph/api/v2/wtcph/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top

Warnings

  • `crawl-delay` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top