worldmart-tokyo.com
robots.txt

Robots Exclusion Standard data for worldmart-tokyo.com

Resource Scan

Scan Details

Site Domain worldmart-tokyo.com
Base Domain worldmart-tokyo.com
Scan Status Ok
Last Scan2024-11-12T05:51:11+00:00
Next Scan 2024-11-19T05:51:11+00:00

Last Scan

Scanned2024-11-12T05:51:11+00:00
URL https://worldmart-tokyo.com/robots.txt
Domain IPs 8.217.135.250
Response IP 8.217.135.250
Found Yes
Hash 6e17cad53329f767a2019ce3aaee7ed9156e4cb0ccdaea23df9813f3910858f1
SimHash 4d5be9f28131

Groups

*

Rule Path
Disallow /yahoo_shopping_category_list/
Disallow /yahoo_shopping_list/
Disallow /yahoo_shopping_detail/
Disallow /ja/yahoo_shopping_category_list/
Disallow /ja/yahoo_shopping_list/
Disallow /ja/yahoo_shopping_detail/
Disallow /en/yahoo_shopping_category_list/
Disallow /en/yahoo_shopping_list/
Disallow /en/yahoo_shopping_detail/
Disallow /zh-cn/yahoo_shopping_category_list/
Disallow /zh-cn/yahoo_shopping_list/
Disallow /zh-cn/yahoo_shopping_detail/
Disallow /zh-tw/yahoo_shopping_category_list/
Disallow /zh-tw/yahoo_shopping_list/
Disallow /zh-tw/yahoo_shopping_detail/
Disallow /rakuten_list/
Disallow /rakuten_category_list/
Disallow /rakuten_detail/
Disallow /ja/rakuten_list/
Disallow /ja/rakuten_category_list/
Disallow /ja/rakuten_detail/
Disallow /en/rakuten_list/
Disallow /en/rakuten_category_list/
Disallow /en/rakuten_detail/
Disallow /zh-cn/rakuten_list/
Disallow /zh-cn/rakuten_category_list/
Disallow /zh-cn/rakuten_detail/
Disallow /zh-tw/rakuten_list/
Disallow /zh-tw/rakuten_category_list/
Disallow /zh-tw/rakuten_detail/

amazonbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 100000000