drinksco.fr
robots.txt

Robots Exclusion Standard data for drinksco.fr

Resource Scan

Scan Details

Site Domain drinksco.fr
Base Domain drinksco.fr
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-10-05T18:18:32+00:00
Next Scan 2024-10-19T18:18:32+00:00

Last Successful Scan

Scanned2023-05-02T11:53:16+00:00
URL https://drinksco.fr/robots.txt
Redirect https://www.drinksco.fr/robots.txt
Redirect Domain www.drinksco.fr
Redirect Base drinksco.fr
Domain IPs 3.248.13.17, 54.74.157.147, 54.76.144.48
Redirect IPs 3.248.13.17, 54.74.157.147, 54.76.144.48
Response IP 54.74.157.147
Found Yes
Hash ec1b8e753dd96bfea1726038b8b1d1b328842e17ada3bb268fd4c74b4999883d
SimHash e314f053eab9

Groups

*

Rule Path
Disallow /header-tabs-ajax
Disallow /connect-authorize
Disallow /connect-callback
Disallow /connect-unlink
Disallow /tracking.json
Disallow /load-module
Disallow /1%3Ak%3A1
Disallow /*_s%3A
Disallow /s$
Disallow /*%2C*%2C
Disallow /*%3As%3A
Disallow /*%3Ah%3A
Disallow /store
Disallow /checkout
Disallow /*/prix%3A
Disallow /*/prix/*
Disallow *search%3D*
Disallow *?opinions*
Disallow *?*maker=
Disallow *?*sparkling_color=
Disallow *?*badges=
Disallow *?*sort=
Disallow *?*price
Disallow /*/2271556/
Disallow /ajax-*
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

lcc

Rule Path
Disallow /

bubing

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

Comments

  • Allow: /*/prix:FR
  • Prevent Google DFP problems.
  • Allow: /
  • http://corpora.informatik.uni-leipzig.de/crawler_faq.html
  • http://law.di.unimi.it/BUbiNG.html