dropwinkel.eu
robots.txt

Robots Exclusion Standard data for dropwinkel.eu

Resource Scan

Scan Details

Site Domain dropwinkel.eu
Base Domain dropwinkel.eu
Scan Status Ok
Last Scan2025-04-29T04:40:43+00:00
Next Scan 2025-05-13T04:40:43+00:00

Last Scan

Scanned2025-04-29T04:40:43+00:00
URL https://dropwinkel.eu/robots.txt
Redirect https://www.dropwinkel.eu/robots.txt
Redirect Domain www.dropwinkel.eu
Redirect Base dropwinkel.eu
Domain IPs 83.96.200.86
Redirect IPs 83.96.200.86
Response IP 83.96.200.86
Found Yes
Hash b5a11e132743f9960e6f4b4c9f5c1ecabec8e7504acc11c2d7d189de59969e29
SimHash 70501901ad32

Groups

*

Rule Path
Disallow /account
Disallow /login
Disallow /register
Disallow /checkout/*
Disallow /search*
Disallow /*?name=
Disallow /*%26name%3D
Disallow /*?brand
Disallow /*%26brand
Disallow /*?price
Disallow /*%26price
Disallow /*?attributes
Disallow /*%26attributes
Disallow /bot-trap
Disallow /roundcube
Disallow /webmail
Disallow /squirrelmail

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /media/

Other Records

Field Value
crawl-delay 10

facebot

Rule Path
Allow /media/

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Allow /

semrushbot-ba

Rule Path
Allow /

semrushbot-si

Rule Path
Allow /

semrushbot-swa

Rule Path
Allow /

semrushbot-ct

Rule Path
Allow /

semrushbot-bm1

Rule Path
Allow /

oai-searchbot
chatgpt-user
perplexitybot
andibot
phindbot
youbot

Rule Path
Allow /

gptbot
ccbot
google-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.dropwinkel.eu/sitemap_index.xml

Comments

  • Algemene restricties voor alle bots
  • Bot-trap
  • Webmail
  • Specifieke bots blokkeren/vertragen
  • SEMRush Bots
  • Allow AI search and agent use
  • Disallow AI training data collection