Log In Sign Up

costco.co.jp
robots.txt

Robots Exclusion Standard data for costco.co.jp

Resource Scan

Scan Details

Site Domain costco.co.jp
Base Domain costco.co.jp
Scan Status Ok
Last Scan2024-10-25T23:15:10+00:00
Next Scan 2024-11-24T23:15:10+00:00

Last Scan

Scanned2024-10-25T23:15:10+00:00
URL https://costco.co.jp/robots.txt
Redirect https://www.costco.co.jp/robots.txt
Redirect Domain www.costco.co.jp
Redirect Base costco.co.jp
Domain IPs 35.192.62.159
Redirect IPs 23.50.82.22
Response IP 23.50.82.22
Found Yes
Hash 16749f2851c2feeef9ec71f0cb9636a09ec068b1156ffbe945324d02213579b3
SimHash 28561f36cdf8

Groups

*

Rule Path
Disallow /checkout
Disallow /my-account
Disallow /c/costco?page=
Disallow /cart/miniCart/SUBTOTAL

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.costco.co.jp/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
Back to top