beps.it
robots.txt

Robots Exclusion Standard data for beps.it

Resource Scan

Scan Details

Site Domain beps.it
Base Domain beps.it
Scan Status Ok
Last Scan2025-10-07T22:10:30+00:00
Next Scan 2025-11-06T22:10:30+00:00

Last Scan

Scanned2025-10-07T22:10:30+00:00
URL https://beps.it/robots.txt
Redirect https://www.beps.it/it/robots.txt
Redirect Domain www.beps.it
Redirect Base beps.it
Domain IPs 104.21.38.92, 172.67.221.122, 2606:4700:3035::6815:265c, 2606:4700:3037::ac43:dd7a
Redirect IPs 104.26.10.164, 104.26.11.164, 172.67.72.135, 2606:4700:20::681a:aa4, 2606:4700:20::681a:ba4, 2606:4700:20::ac43:4887
Response IP 172.67.72.135
Found Yes
Hash 9b82e708d3852ec15b7f54f9058d74f85b5e6bd6e8e6e587e98b997c738f4066
SimHash 2270dd5241f3

Groups

*

Rule Path
Disallow /*?
Allow /*?p=
Disallow /index.php/
Disallow /wishlist
Disallow /admin/
Disallow /sendfriend/
Disallow /enable-cookies/
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /skin/
Disallow /js/
Disallow /directory/
Allow /*?utm*
Disallow /*checkout/
Disallow /*onestepcheckout/
Disallow /*customer/
Disallow /*catalogsearch/
Disallow /*catalog/product_compare/
Disallow /*catalog/category/view/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /tag/
Disallow /*/tag/
Disallow /review/
Disallow /it/shoponline
Disallow /it/review/product/listAjax/
Disallow /it/catene-neve/*size%3D
Disallow /it/catene-neve/*year%3D
Allow /it/catene-neve/fiat/*year%3D
Allow /it/catene-neve/jeep/*year%3D
Allow /it/catene-neve/renault/*year%3D
Allow /*html?attribute151=
Disallow /it/barre-portatutto-portapacchi/*version%3D

Comments

  • Allow: /*?special_label=8463$
  • Stop crawling user account and checkout pages
  • Blocking native catalog and search pages:
  • Sometimes Webmasters block pages with filters..
  • More reasonable to use canonical tag on these pages.
  • Blocking CMS directories.
  • These commands are not necessary. Search engines are smart enough to avoid including CMS files in their index.
  • Blocking duplicate content: