de.openfoodfacts.org
robots.txt

Robots Exclusion Standard data for de.openfoodfacts.org

Resource Scan

Scan Details

Site Domain de.openfoodfacts.org
Base Domain openfoodfacts.org
Scan Status Ok
Last Scan2024-09-22T18:46:20+00:00
Next Scan 2024-10-06T18:46:20+00:00

Last Scan

Scanned2024-09-22T18:46:20+00:00
URL https://de.openfoodfacts.org/robots.txt
Domain IPs 213.36.253.214
Response IP 213.36.253.214
Found Yes
Hash b0fda40020487e1939fb548077aa2a3dfae2bf443b2745a7c61dc1ff2378888c
SimHash 525c8698cf40

Groups

*

Rule Path
Allow /cgi/product_image.pl
Allow /cgi/opensearch.pl
Disallow /cgi
Disallow /code
Disallow /api
Disallow /additives
Disallow /zusatzstoffe
Disallow /allergen
Disallow /allergens
Disallow /allergene
Disallow /amino-acid
Disallow /amino-acids
Disallow /aminosaure
Disallow /aminosauren
Disallow /brands
Disallow /marken
Disallow /categories
Disallow /kategorien
Disallow /categories-properties
Disallow /checker
Disallow /checkers
Disallow /prufer
Disallow /city
Disallow /cities
Disallow /stadt
Disallow /staedte
Disallow /code
Disallow /codes
Disallow /corrector
Disallow /correctors
Disallow /korrekteur
Disallow /korrektoren
Disallow /country
Disallow /countries
Disallow /land
Disallow /lander
Disallow /data-quality
Disallow /data-quality-bug
Disallow /data-quality-bugs
Disallow /data-quality-error
Disallow /data-quality-errors
Disallow /data-quality-error-producers
Disallow /data-quality-errors-producers
Disallow /data-quality-info
Disallow /data-quality-warning
Disallow /data-quality-warnings
Disallow /data-quality-warning-producers
Disallow /data-quality-warnings-producers
Disallow /data-source
Disallow /data-sources
Disallow /datenquelle
Disallow /datenquellen
Disallow /debug
Disallow /editor
Disallow /editors
Disallow /editoren
Disallow /packager-code
Disallow /packager-codes
Disallow /produzenten-code
Disallow /produzenten-codes
Disallow /entry-date
Disallow /entry-dates
Disallow /datum-der-eintragung
Disallow /datum-der-eintragungen
Disallow /food-group
Disallow /food-groups
Disallow /import
Disallow /imports
Disallow /possible-improvement
Disallow /possible-improvements
Disallow /informer
Disallow /informers
Disallow /informant
Disallow /informanten
Disallow /ingredient
Disallow /ingredients
Disallow /zutat
Disallow /zutaten
Disallow /ingredients-analysis
Disallow /analyse-der-inhaltsstoffe
Disallow /ingredients-from-palm-oil
Disallow /zutat-aus-palmol
Disallow /zutaten-aus-palmol
Disallow /number-of-ingredients
Disallow /numbers-of-ingredients
Disallow /anzahl-der-zutaten
Disallow /anzahlen-der-zutaten
Disallow /ingredient-original
Disallow /ingredients-original
Disallow /ingredients-that-may-be-from-palm-oil
Disallow /zutat-die-moglicherweise-aus-palmol-hergestellt-wird
Disallow /inhaltsstoffe-die-moglicherweise-aus-palmol-hergestellt-werden
Disallow /known-nutrient
Disallow /known-nutrients
Disallow /bekannter-nahrstoff
Disallow /bekannte-nahrstoffe
Disallow /labels
Disallow /language
Disallow /languages
Disallow /sprache
Disallow /sprachen
Disallow /last-check-date
Disallow /last-check-dates
Disallow /datum-der-letzten-prufung
Disallow /daten-der-letzten-prufungen
Disallow /last-edit-date
Disallow /last-edit-dates
Disallow /datum-der-letzten-bearbeitung
Disallow /datum-der-letzten-bearbeitungen
Disallow /last-image-date
Disallow /last-image-dates
Disallow /datum-vom-letzten-foto
Disallow /datum-der-letzten-fotos
Disallow /manufacturing-place
Disallow /manufacturing-places
Disallow /herstellungsort
Disallow /herstellungsorte
Disallow /mineral
Disallow /minerals
Disallow /mineralie
Disallow /mineralien
Disallow /misc
Disallow /sonstiges
Disallow /mission
Disallow /missions
Disallow /aufgabe
Disallow /aufgaben
Disallow /nova-groups
Disallow /nova-gruppen
Disallow /nucleotide
Disallow /nucleotides
Disallow /nukleotid
Disallow /nukleotide
Disallow /nutrient-level
Disallow /nutrient-levels
Disallow /nahrwert-stufe
Disallow /nahrwert-stufen
Disallow /nutrient
Disallow /nutrients
Disallow /nahrstoff
Disallow /nahrstoffe
Disallow /nutri-score
Disallow /nutri-score-2021
Disallow /nutri-score-2023
Disallow /nutrition-grades
Disallow /naehrwertqualitaet
Disallow /origin
Disallow /origins
Disallow /herkunft
Disallow /herkunfte
Disallow /other-nutritional-substance
Disallow /other-nutritional-substances
Disallow /sonstiger-zugesetzter-nahrstoff
Disallow /sonstige-zugesetzte-nahrstoffe
Disallow /owner
Disallow /owners
Disallow /packaging
Disallow /verpackung
Disallow /verpackungen
Disallow /packaging-materials
Disallow /packaging-recycling
Disallow /packaging-shape
Disallow /packaging-shapes
Disallow /period-after-opening
Disallow /periods-after-opening
Disallow /zeitraum-nach-dem-offnen
Disallow /zeitraume-nach-dem-offnen
Disallow /photographer
Disallow /photographers
Disallow /fotograf
Disallow /fotografen
Disallow /pnns-group-1
Disallow /pnns-groups-1
Disallow /pnns-gruppe-1
Disallow /pnns-gruppen-1
Disallow /pnns-group-2
Disallow /pnns-groups-2
Disallow /pnns-gruppe-2
Disallow /pnns-gruppen-2
Disallow /popularity
Disallow /purchase-place
Disallow /purchase-places
Disallow /verkaufsort
Disallow /verkaufsorte
Disallow /quality
Disallow /qualitat
Disallow /state
Disallow /states
Disallow /status
Disallow /store
Disallow /stores
Disallow /geschaft
Disallow /geschafte
Disallow /team
Disallow /teams
Disallow /trace
Disallow /traces
Disallow /spur
Disallow /spuren
Disallow /unknown-nutrient
Disallow /unknown-nutrients
Disallow /unbekannter-nahrwert
Disallow /unbekannte-n%C3%A4hrwerte
Disallow /contributor
Disallow /contributors
Disallow /mitwirkende
Disallow /mitwirkenden
Disallow /vitamin
Disallow /vitamins
Disallow /vitamine
Disallow /weigher
Disallow /weighers

bingbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

cliqzbot/3.0

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seekport bot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

scrapy/1.5.0

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

velenpublicwebcrawler (velen.io)

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

semrushbot/2~bl

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

yandexmarket

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

Comments

  • Disallow: Bingbot (temporary test)
  • Disallow: SEOkicks-Robot
  • http://www.opensiteexplorer.org/dotbot
  • http://www.searchmetrics.com
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://www.domaintools.com/webmasters/surveybot.php
  • http://www.seodiver.com/bot
  • http://openlinkprofiler.org/bot
  • http://www.wotbox.com/bot/