mk16.de
robots.txt

Robots Exclusion Standard data for mk16.de

Resource Scan

Scan Details

Site Domain mk16.de
Base Domain mk16.de
Scan Status Ok
Last Scan2024-10-09T16:57:09+00:00
Next Scan 2024-10-16T16:57:09+00:00

Last Scan

Scanned2024-10-09T16:57:09+00:00
URL https://mk16.de/robots.txt
Domain IPs 2a03:4000:6:2784::1, 5.45.109.122
Response IP 5.45.109.122
Found Yes
Hash dd27b2f066b0ebbcd8755cc7661540acbf791203861fc0a187c76670035fccc6
SimHash f05e1b51b4a0

Groups

diffbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

awakari

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrush

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

pingdom

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

googleother

Rule Path
Disallow /

googleother-image

Rule Path
Disallow /

googleother-video

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

*

Rule Path
Disallow /404.html
Disallow /blog/bgp-communities/
Disallow /blog/gl-inet-travel-router-comparison/
Disallow /blog/mikrotik-dn42/
Disallow /blog/ntp-servers/
Allow /

Other Records

Field Value
sitemap /sitemap.xml

Warnings

  • 2 invalid lines.