shopee.com.my
robots.txt
Robots Exclusion Standard data for shopee.com.my
Resource Scan
Scan Details
| Site Domain | shopee.com.my |
| Base Domain | shopee.com.my |
| Scan Status | Ok |
| Last Scan | 2024-11-13T21:56:30+00:00 |
| Next Scan | 2024-11-20T21:56:30+00:00 |
Last Scan
| Scanned | 2024-11-13T21:56:30+00:00 |
| URL | https://shopee.com.my/robots.txt |
| Domain IPs | 143.92.75.82 |
| Response IP | 202.181.90.194 |
| Found | Yes |
| Hash | 476b071155013b86d521ecf2afd6374294da23abc31533ec8e99eca755dd20b0 |
| SimHash | 7610c216798c |
Groups
googlebot
googlebot-mobile
bingbot
| Rule | Path |
|---|---|
| Disallow | /cart/ |
| Disallow | /checkout/ |
| Disallow | /buyer/login/otp |
| Disallow | /user/ |
| Disallow | /me/ |
| Disallow | /order/ |
| Disallow | /daily_discover/ |
| Disallow | /mall/just-for-you/ |
| Disallow | /mall/*-cat. |
| Disallow | /from_same_shop/ |
| Disallow | /you_may_also_like/ |
| Disallow | *-i.*/similar?from=flash_sale |
| Disallow | /find_similar_products/ |
| Disallow | /top_products |
| Disallow | /search*searchPrefill |
| Disallow | /index.html |
| Disallow | /search*hashtag%3D |
| Disallow | /search?shop= |
| Disallow | /buyer/*/rating/ |
| Disallow | *-cat.*?category= |
| Disallow | /shop/*/search |
| Disallow | /shop/*/details |
| Disallow | /search/? |
| Disallow | /search?brands= |
| Disallow | /search?originalCategoryId= |
| Disallow | /addon-deal-selection/ |
| Disallow | /bundle-deal/ |
| Disallow | /*?*sp_atk= |
| Disallow | *__classic__%3D1 |
| Disallow | /search_user |
| Disallow | /blog/?s= |
| Disallow | *utm_source |
| Disallow | /search*.com |
| Disallow | /search*%C3%82%C2%B7asia |
| Disallow | /search*.tv |
| Disallow | /search*.xyz |
| Disallow | /search*%C3%82%C2%B7COM |
| Disallow | /search*%C3%A5%C2%A6%C2%B9 |
| Disallow | /search*%C3%A9 |
| Disallow | /search*%C3%A5 |
| Disallow | /*?*srsltid |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.1 |
*
| Rule | Path |
|---|---|
| Disallow | /cart/ |
| Disallow | /checkout/ |
| Disallow | /buyer/login/otp |
| Disallow | /user/ |
| Disallow | /me/ |
| Disallow | /order/ |
| Disallow | /daily_discover/ |
| Disallow | /mall/just-for-you/ |
| Disallow | /mall/*-cat. |
| Disallow | /from_same_shop/ |
| Disallow | /you_may_also_like/ |
| Disallow | *-i.*/similar |
| Disallow | /find_similar_products/ |
| Disallow | /top_products |
| Disallow | /search*searchPrefill |
| Disallow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Warnings
- 29 invalid lines.