xxl.dk
robots.txt

Robots Exclusion Standard data for xxl.dk

Resource Scan

Scan Details

Site Domain xxl.dk
Base Domain xxl.dk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-03T16:22:07+00:00
Next Scan 2024-06-17T16:22:07+00:00

Last Successful Scan

Scanned2024-04-26T16:21:25+00:00
URL https://xxl.dk/robots.txt
Redirect https://www.xxl.dk/robots.txt
Redirect Domain www.xxl.dk
Redirect Base xxl.dk
Domain IPs 65.9.112.110, 65.9.112.27, 65.9.112.51, 65.9.112.95
Redirect IPs 13.226.2.52, 13.226.2.6, 13.226.2.89, 13.226.2.98, 2600:9000:200f:4c00:b:9f65:f280:93a1, 2600:9000:200f:5000:b:9f65:f280:93a1, 2600:9000:200f:8800:b:9f65:f280:93a1, 2600:9000:200f:8c00:b:9f65:f280:93a1, 2600:9000:200f:9a00:b:9f65:f280:93a1, 2600:9000:200f:a000:b:9f65:f280:93a1, 2600:9000:200f:a00:b:9f65:f280:93a1, 2600:9000:200f:be00:b:9f65:f280:93a1
Response IP 3.160.246.5
Found Yes
Hash e7581c525bba3cafe0dfb1c8901b7b7f67453f7550075d3206b1283b32711c0c
SimHash c478fd9bcffe

Groups

*

Rule Path
Disallow /account
Disallow /cart
Disallow /checkout
Disallow /login
Disallow /search
Disallow *?*Hjulst%C3%B8rrelse=*
Disallow *?*Pronation=*
Disallow *?*Age=*
Disallow *?*Motor%2BSensor=*
Disallow *?*Frame%2BMaterial=*
Disallow *?*Groupset=*
Disallow *?*Brake%2Bsystem=*
Disallow *?*Number%2Bof%2Bgears=*
Disallow *?*Motor%2BPlacement=*
Disallow *?*Battery%2BCapacity=*
Disallow *?*Motor%2BVendor=*
Disallow *?*Type%2BCyclingshoes=*
Disallow *?*GPS=*
Disallow *?*Varehuse=*
Disallow *?*Kampagne=*
Disallow *?*Skitype=*
Disallow *?*Userlevel=*
Disallow *?*Terrain=*
Disallow *?*Base=*
Disallow *?*Ski%2Bwidth=*
Disallow *?*Bindings=*
Disallow *?*Material=*
Disallow *?*Telescopic%2Bpole=*
Disallow *?*Condition=*
Disallow *?*Wax=*
Disallow *?*Flourcontain=*
Disallow *?*Type%2Bof%2Bwax=*
Disallow *?*Kampagne=*
Disallow *?*Fork%2BType=*
Disallow *?*Bike%2BLocks=*
Disallow *?*Trainer%2BModel=*
Disallow *?*Sleepingbags%2BTemp=*
Disallow *?*Fuel=*
Disallow *?*Weight=*
Disallow *?*Water%2Btype=*
Disallow *?*Shotgun%2BCaliber=*
Disallow *?*Lead%2B%2F%2BNon-lead=*
Disallow *?*Shotgun%2BMec=*
Disallow *?*Width%2Bof%2Bscope=*
Disallow *?*Rifle%2BCaliber=*
Disallow *?*Serie=*
Disallow *?*NHL%2BHold=*
Disallow *?*Type%2BGolfclubs=*
Disallow *?*V%C3%A5ddragt%2Btykkelse=*
Disallow *?*Fatning=*
Disallow *?*V%C3%A5ddragt%2Bkort%2Flang=*
Disallow *?*Type%2BGolfsets=*
Disallow *?*Golfbolde=*
Disallow *?*Snorkling=*
Disallow *?*V%C3%A5ddragt=*
Disallow *?*Vests=*
Disallow *?*Pris=*
Disallow *?*Kategori=*
Disallow *?*Category=*
Disallow *?*Style%2BSwatch=*

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

*

Rule Path
Disallow /awesomeproduct

Comments

  • For all robots
  • Block access to specific groups of pages
  • Block access to search results
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot
  • Block chinese, korean and russian bots