boots.no
robots.txt

Robots Exclusion Standard data for boots.no

Resource Scan

Scan Details

Site Domain boots.no
Base Domain boots.no
Scan Status Ok
Last Scan2024-09-29T13:48:54+00:00
Next Scan 2024-10-29T13:48:54+00:00

Last Scan

Scanned2024-09-29T13:48:54+00:00
URL https://boots.no/robots.txt
Redirect https://www.boots.no/robots.txt
Redirect Domain www.boots.no
Redirect Base boots.no
Domain IPs 20.8.56.197
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash 4accc08a60784b5c41d353e89dd360ff16e2a286cede09d3073a96a2a3e4fd93
SimHash 6771c753ccf8

Groups

googlebot-image

Rule Path
Allow /media/catalog/product/

*

Rule Path
Allow /*static/version*/*.css
Allow /*static/version*/*.svg
Disallow /index.php/
Disallow /*?
Disallow /checkout/
Disallow /app/
Disallow /lib/
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sales/
Disallow /sendfriend/
Disallow /wishlist/
Disallow /review/
Disallow /banner/ajax/
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*?dir=
Disallow /*?*p=
Disallow /*?*q=
Disallow /*?*price=
Disallow /*.php$
Disallow /*SID%3D
Disallow /*static/version*
Disallow /pso/
Disallow /kco/api/

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.boots.no/media/sitemap/sitemap_index.xml

Comments

  • For all robots
  • Enable static files for mobile search
  • Block access to specific groups of pages
  • Do not index pages that are sorted or filtered.
  • Do not index session ID
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot