Log In Sign Up

hopt.nl
robots.txt

Robots Exclusion Standard data for hopt.nl

Resource Scan

Scan Details

Site Domain hopt.nl
Base Domain hopt.nl
Scan Status Ok
Last Scan2024-10-24T01:08:18+00:00
Next Scan 2024-11-23T01:08:18+00:00

Last Scan

Scanned2024-10-24T01:08:18+00:00
URL https://hopt.nl/robots.txt
Redirect https://www.hopt.nl/robots.txt
Redirect Domain www.hopt.nl
Redirect Base hopt.nl
Domain IPs 2600:9000:2024:3c00:2:b255:7d40:93a1, 2600:9000:2024:6a00:2:b255:7d40:93a1, 2600:9000:2024:7200:2:b255:7d40:93a1, 2600:9000:2024:9200:2:b255:7d40:93a1, 2600:9000:2024:a200:2:b255:7d40:93a1, 2600:9000:2024:d000:2:b255:7d40:93a1, 2600:9000:2024:de00:2:b255:7d40:93a1, 2600:9000:2024:f000:2:b255:7d40:93a1, 65.9.112.22, 65.9.112.44, 65.9.112.52, 65.9.112.74
Redirect IPs 13.227.254.66, 13.227.254.74, 13.227.254.75, 13.227.254.89, 2600:9000:200a:1a00:1e:484f:61c0:93a1, 2600:9000:200a:2600:1e:484f:61c0:93a1, 2600:9000:200a:6000:1e:484f:61c0:93a1, 2600:9000:200a:a00:1e:484f:61c0:93a1, 2600:9000:200a:b200:1e:484f:61c0:93a1, 2600:9000:200a:de00:1e:484f:61c0:93a1, 2600:9000:200a:f200:1e:484f:61c0:93a1, 2600:9000:200a:fe00:1e:484f:61c0:93a1
Response IP 13.227.254.66
Found Yes
Hash 0fc7b8429c7b9b16051d8d6334e8ed205666f2c89588f60762ffd5d9419a779b
SimHash 584073d3b8bc

Groups

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pkginfo/
Disallow /report/
Disallow /setup/
Disallow /update/
Disallow /var/
Disallow /vendor/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$
Disallow */dimensione-*
Disallow */inhoud-*
Disallow */gradazione-*
Disallow */alcoholpercentage-*
Disallow */manufacturer-*
Disallow */glasvorm-*
Disallow */fust_soort-*
Disallow */fut_type-*
Disallow */o-date
Disallow */o-rate
Disallow */o-price
Disallow */search-result/*
Disallow */frontend*
Disallow /fr/3-biere-bouteille/pays-*-*
Disallow /en/3-bottled-beer/country-*-*
Disallow /fr/2-*
Disallow /en/2-*
Disallow /2-*
Disallow */forgot-password/*
Disallow */reset-password/*
Disallow */reset-password-new/*
Disallow */give/*
Disallow /fr/fiche-brasserie.html
Disallow /en-gb/*?*addtocart=
Disallow /en-gb/*?*addtocarterror=
Disallow /en-gb/*?*sortBy=
Disallow /fr/741-les-offres-leffe
Disallow /fr/780-selection-bieres-d-ete
Disallow /en/780-selection-of-summer-beers
Disallow /780-*
Disallow /fr/lpn/214-saint-patrick
Disallow /nl-be/flessen
Disallow /nl-be/3-flessen
Disallow /nl-be/70-bierpakketten
Disallow /nl-be/494-thuisbrouwpakket
Disallow /nl-be/518-hop
Disallow /nl-be/526-gist
Disallow /nl-be/707-big-packs
Disallow /nl-be/thuisbrouwpakket
Disallow /nl-be/500-all-grain-bier-kit
Disallow /nl-be/664-beerkits-navullingen
Disallow /nl-be/237-hop-om-bier-te-brouwen
Disallow /nl-be/63-keykeg-biertap
Disallow /nl-be/611-bieren-van-de-beery-christmas-2017
Disallow /de-de/3-flaschen-bier
Disallow /de-de/flaschen-bier
Disallow /de-de/707-bier-grossverpackung
Disallow /nl-nl/flessen
Disallow /nl-nl/3-flessen
Disallow /nl-nl/70-bierpakketten
Disallow /nl-nl/bierpakketten
Disallow /nl-nl/494-thuisbrouwpakket
Disallow /nl-nl/518-hop
Disallow /nl-nl/526-gist
Disallow /nl-nl/thuisbrouwpakket
Disallow /nl-nl/500-all-grain-bier-kit
Disallow /nl-nl/707-big-packs
Disallow /nl-nl/664-beerkits-navullingen
Disallow /nl-nl/237-hop-om-bier-te-brouwen
Disallow /nl-nl/611-bieren-van-de-beery-christmas-2017
Disallow /it-it/bottiglie
Disallow /it-it/3-bottiglie
Disallow /it-it/70-cofanetti-di-birra-artigianale
Disallow /it-it/494-kit-di-brassaggio
Disallow /it-it/cofanetti-di-birra-artigianale
Disallow /it-it/707-confezioni-risparmio
Disallow /it-it/kit-di-brassaggio
Disallow /it-it/526-lieviti
Disallow /it-it/664-kit-di-brassaggio-e-ricariche
Disallow /it-it/524-malti
Disallow /it-it/518-luppoli
Disallow /it-it/237-luppoli-per-il-brassaggio
Disallow /it-it/500-kit-ricette-per-tutti-i-grani
Disallow /it-it/235-malti-per-il-brassaggio
Disallow /it-it/confezioni-risparmio
Disallow /es-es/botellas
Disallow /es-es/3-botellas
Disallow /es-es/70-pack-de-cervezas-artesanales
Disallow /es-es/518-lupulos
Disallow /es-es/494-kit-de-elaboracion-de-cerveza
Disallow /es-es/237-lupulos
Disallow /es-es/526-levaduras
Disallow /es-es/lupulos
Disallow /es-es/pack-de-cervezas-artesanales
Disallow /fr-be/3-bouteilles
Disallow /fr-be/17-echanges-points-cadeaux
Disallow /fr-be/bouteilles
Disallow /fr-be/623-b2b
Disallow /fr-be/88-pour-350-points
Disallow /sv-se/3-flaskor
Disallow /sv-se/70-samlingar
Disallow /sv-se/707-big-packs
Disallow /sv-se/samlingar
Disallow /sv-se/flaskor
Disallow /sv-se/493-olbryggning
Disallow /sv-se/774-mega-packs
Disallow /sv-se/664-olkit-receptkit
Disallow /sv-se/753-all-big-packs
Disallow /en-gb/index.php/
Disallow /en-gb/catalog/product_compare/
Disallow /en-gb/catalog/category/view/
Disallow /en-gb/catalog/product/view/
Disallow /en-gb/catalogsearch/
Disallow /en-gb/search/
Disallow /en-gb/checkout/
Disallow /en-gb/control/
Disallow /en-gb/contacts/
Disallow /en-gb/customer/
Disallow /en-gb/reward/customer/
Disallow /en-gb/keg/returns/
Disallow /en-gb/sales/order/
Disallow /en-gb/customize/
Disallow /en-gb/newsletter/
Disallow /en-gb/review/
Disallow /en-gb/sendfriend/
Disallow /en-gb/wishlist/
Disallow /en-gb/test/
Disallow /en-gb/beerhawk_wishlist
Disallow /en-gb/nosto-rw
Disallow /en-gb/trade
Disallow /en-gb/kegreturns
Disallow /en-gb/beer-hawk-craft-favourites-mixed-6-pack
Disallow /en-gb/beer-hawk-world-lager-mixed-6-pack
Disallow /en-gb/hoegaarden-0-0-alcohol-free-24-beers
Disallow /en-gb/catalog/product/gallery/*
Disallow /en-gb/huggg
Disallow /en-gb/huggg-level-2
Disallow /en-gb/monzo-beer-hawk-beery-gift-box
Disallow /en-gb/bfprize-mon-kjbga2nk
Disallow /en-gb/bfprize-tue-ljb2ajlf
Disallow /en-gb/bfprize-wed-j2an5l3n
Disallow /en-gb/bfprize-thurs-8ia9fbds
Disallow /en-gb/bfprize-fri-pas8bjef
Disallow /en-gb/perfectdraft-budweiser-starter-bundle-cp
Disallow /en-gb/add/to/cart*
Disallow /en-gb/privacy-policy-v1-2016
Disallow /en-gb/privacy-policy-v2-2018
Disallow /en-gb/privacy-policy-v3-2021
Disallow /en-gb/misc-instalment-category
Disallow /en-gb/gift-vouchers-category
Disallow /en-gb/alkngdf-clearpay-exclusion
Disallow /en-gb/perfectdraft-pro-bundle
Disallow /en-gb/app-keg-listing
Disallow /en-gb/perfect-draft-range/app-keg-listing

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

adidxbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

buenibot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /
Disallow /
Disallow /

bingbot
applebot

Rule Path
Disallow /*-*/*
Disallow /nl-be/*-*/*
Disallow /fr-be/*-*/*
Disallow /de-de/*-*/*
Disallow /nl-nl/*-*/*
Disallow /es-es/*-*/*
Disallow /it-it/*-*/*
Disallow /sv-se/*-*/*
Allow /*-*/*.html
Allow /nl-be/*-*/*.html
Allow /fr-be/*-*/*.html
Allow /de-de/*-*/*.html
Allow /nl-nl/*-*/*.html
Allow /es-es/*-*/*.html
Allow /it-it/*-*/*.html
Allow /sv-se/*-*/*.html
Disallow /*_*/*
Disallow /nl-be/*_*/*
Disallow /fr-be/*_*/*
Disallow /de-de/*_*/*
Disallow /nl-nl/*_*/*
Disallow /es-es/*_*/*
Disallow /it-it/*_*/*
Disallow /sv-se/*_*/*
Allow /magazine/*
Allow /nl-be/magazine/*
Allow /fr-be/magazine/*
Allow /de-de/magazine/*
Allow /nl-nl/magazine/*
Allow /es-es/magazine/*
Allow /it-it/magazine/*
Allow /sv-se/magazine/*

googlebot

Rule Path
Disallow /nl-be/catalogsearch/
Disallow /fr-be/catalogsearch/
Disallow /de-de/catalogsearch/
Disallow /nl-nl/catalogsearch/
Disallow /es-es/catalogsearch/
Disallow /it-it/catalogsearch/
Disallow /sv-se/catalogsearch/
Disallow /en-gb/catalogsearch/

googlebot-image

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.perfectdraft.com/en-gb/blog_sitemap.xml
sitemap https://www.perfectdraft.com/en-gb/sitemap.xml
Back to top

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • shop 8
  • Directories
  • Files
  • CVS, SVN directory and dump files
  • Avoid undesired crawlers
  • Specific rules for bingbot and applebot to disallow cats and brands with filters but allow product categories
  • Category listing pages
  • Product pages
  • Manufacturer listing page
  • Blog pages
  • Google Feed Fix
  • Sitemaps
Back to top

Warnings

  • 2 invalid lines.
  • `useragent` is not a known field.
Back to top