drinksco.nl
robots.txt

Robots Exclusion Standard data for drinksco.nl

Resource Scan

Scan Details

Site Domain drinksco.nl
Base Domain drinksco.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonRequest timed out.
Last Scan2024-08-07T11:20:42+00:00
Next Scan 2024-11-05T11:20:42+00:00

Last Successful Scan

Scanned2023-04-04T15:11:43+00:00
URL https://drinksco.nl/robots.txt
Redirect https://www.drinksco.nl/robots.txt
Redirect Domain www.drinksco.nl
Redirect Base drinksco.nl
Domain IPs 54.194.18.152, 54.220.188.48, 54.74.254.77
Redirect IPs 54.194.18.152, 54.220.188.48, 54.74.254.77
Response IP 54.194.18.152
Found Yes
Hash 31ab780d6f5f50223cdbc0001bd6615ddc007fc1c2056cf97a0c7b61215df635
SimHash e314e8437abb

Groups

*

Rule Path
Disallow /header-tabs-ajax
Disallow /connect-authorize
Disallow /connect-callback
Disallow /connect-unlink
Disallow /tracking.json
Disallow /load-module
Disallow /1%3Ak%3A1
Disallow /*_s%3A
Disallow /s$
Disallow /*%2C*%2C
Disallow /*%3As%3A
Disallow /*%3Ah%3A
Disallow /store
Disallow /checkout
Disallow /*/prijs%3A
Disallow /*/prijs/*
Disallow *search%3D*
Disallow *?opinions*
Disallow *?*maker=
Disallow *?*sparkling_color=
Disallow *?*badges=
Disallow *?*sort=
Disallow *?*price
Disallow /*/2271556/
Disallow /ajax-*
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

lcc

Rule Path
Disallow /

bubing

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

Comments

  • Allow: /*/prijs:NL
  • Prevent Google DFP problems.
  • Allow: /
  • http://corpora.informatik.uni-leipzig.de/crawler_faq.html
  • http://law.di.unimi.it/BUbiNG.html