Log In Sign Up

8020.net
robots.txt

Robots Exclusion Standard data for 8020.net

Resource Scan

Scan Details

Site Domain 8020.net
Base Domain 8020.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-14T08:54:02+00:00
Next Scan 2024-09-12T08:54:02+00:00

Last Successful Scan

Scanned2024-01-24T08:51:26+00:00
URL https://8020.net/robots.txt
Domain IPs 104.26.12.125, 104.26.13.125, 172.67.72.161, 2606:4700:20::681a:c7d, 2606:4700:20::681a:d7d, 2606:4700:20::ac43:48a1
Response IP 104.26.13.125
Found Yes
Hash 66d7fb27a8a39c929243a2acad341d10d6ee80c3cf701401369e450e247fe355
SimHash ab08712fc5c3

Groups

*

Rule Path
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalogsearch/
Disallow /control/
Disallow /contacts/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /sendfriend/
Disallow /wishlist/
Disallow /review/product/listAjax/id/
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*.php$
Disallow /*?SID=
Back to top

Other Records

Field Value
sitemap https://8020.net/sitemaps/sitemap.xml
Back to top

Comments

  • Paths (clean URLs)
  • Need to allow these urls paths to be indexed, since M2 commonly includes these links in sitemap.xml
  • Disallow: /catalog/category/view/
  • Disallow: /catalog/product/view/
  • Do not index pages that are sorted or filtered.
  • Do not index session ID
Back to top