xxl.no
robots.txt

Robots Exclusion Standard data for xxl.no

Resource Scan

Scan Details

Site Domain xxl.no
Base Domain xxl.no
Scan Status Ok
Last Scan2024-05-05T20:12:03+00:00
Next Scan 2024-06-04T20:12:03+00:00

Last Scan

Scanned2024-05-05T20:12:03+00:00
URL https://xxl.no/robots.txt
Redirect https://www.xxl.no/robots.txt
Redirect Domain www.xxl.no
Redirect Base xxl.no
Domain IPs 3.248.44.250, 54.74.127.24
Redirect IPs 13.226.2.124, 13.226.2.28, 13.226.2.47, 13.226.2.55, 2600:9000:24da:1e00:4:838f:3100:93a1, 2600:9000:24da:2000:4:838f:3100:93a1, 2600:9000:24da:2800:4:838f:3100:93a1, 2600:9000:24da:6400:4:838f:3100:93a1, 2600:9000:24da:6600:4:838f:3100:93a1, 2600:9000:24da:a400:4:838f:3100:93a1, 2600:9000:24da:ca00:4:838f:3100:93a1, 2600:9000:24da:d000:4:838f:3100:93a1
Response IP 18.165.171.54
Found Yes
Hash bd7234848ec97a59bff6847d1a7605c89c7dd167ce8e5ac1eaf62d20feee780b
SimHash b475d71ecdea

Groups

*

Rule Path
Disallow /account
Disallow /cart
Disallow /checkout
Disallow /login
Disallow /search
Disallow *?*Hjulst%C3%B8rrelse=*
Disallow *?*Type%2Bsko=*
Disallow *?*Alder=*
Disallow *?*Rammemateriale=*
Disallow *?*Gruppesett=*
Disallow *?*Bremsesystem=*
Disallow *?*Type%2BSykkelsko=*
Disallow *?*GPS=*
Disallow *?*Varehus=*
Disallow *?*Kampanje=*
Disallow *?*Campaign=*
Disallow *?*Campaign=*
Disallow *?*Skitype=*
Disallow *?*Niv%C3%A5=*
Disallow *?*Terreng=*
Disallow *?*S%C3%A5le=*
Disallow *?*Bindningssystem=*
Disallow *?*Ski%2Bwidth=*
Disallow *?*Bindings=*
Disallow *?*G%C3%A5funksjon=*
Disallow *?*Material=*
Disallow *?*Telescopic%2Bpole=*
Disallow *?*F%C3%B8re=*
Disallow *?*Type%2BSm%C3%B8reprodukt=*
Disallow *?*Flourinnhold=*
Disallow *?*Type%2Bof%2Bwax=*
Disallow *?*Kampanje=*
Disallow *?*Gaffel=*
Disallow *?*Sykkell%C3%A5s=*
Disallow *?*Sykkelrulle=*
Disallow *?*Sovepose%2BTemp=*
Disallow *?*Brensel=*
Disallow *?*Vekt=*
Disallow *?*Vekt=*
Disallow *?*Type%2Bvann=*
Disallow *?*Hagle%2BKaliber=*
Disallow *?*Bly%2FIkke%2Bbly=*
Disallow *?*Hagle%2BMec=*
Disallow *?*Diameter%2Bp%C3%A5%2Bmellomr%C3%B8ret=*
Disallow *?*Rifle%2BKaliber=*
Disallow *?*Serie=*
Disallow *?*NHL%2BLag=*
Disallow *?*Type%2BGolfklubber=*
Disallow *?*V%C3%A5tdrakt%2Btykkelse=*
Disallow *?*Fatning=*
Disallow *?*V%C3%A5tdrakt%2Bkort%2Fl%C3%A5ng=*
Disallow *?*Type%2BGolfsett=*
Disallow *?*Golfballer=*
Disallow *?*Snorkling=*
Disallow *?*V%C3%A5tdrakt=*
Disallow *?*Redningsvest=*
Disallow *?*Pris=*
Disallow *?*Kategori=*
Disallow *?*Kategori=*
Disallow *?*Style%2BSwatch=*

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

*

Rule Path
Disallow /awesomeproduct

Comments

  • For all robots
  • Block access to specific groups of pages
  • Block access to search results
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block chinese, korean and russian bots