denniskunkel.com
robots.txt
Robots Exclusion Standard data for denniskunkel.com
Resource Scan
Scan Details
| Site Domain | denniskunkel.com |
| Base Domain | denniskunkel.com |
| Scan Status | Ok |
| Last Scan | 2024-09-13T20:11:08+00:00 |
| Next Scan | 2024-10-13T20:11:08+00:00 |
Last Scan
| Scanned | 2024-09-13T20:11:08+00:00 |
| URL | https://denniskunkel.com/robots.txt |
| Domain IPs | 144.76.242.34 |
| Response IP | 144.76.242.34 |
| Found | Yes |
| Hash | 3c013f922ba0209c2ffcc0528da896aba483e22f33e46176c778d43ef75f01be |
| SimHash | 57004fd0eca0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /Merchant2 |
| Disallow | *.php |
| Disallow | *.asp |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
adbeat_bot
adsbot
ahc
ahrefsbot
aihitbot
aiohttp
amazonadbot
amazonbot
anthropic-ai
applebot-extended
awariobot
awariorssbot
awariosmartbot
barkrowler
blexbot
brandverity
buck
ccbot
chatglm-spider
chatgpt-user
cincraw
claudebot
claude-web
cohere-ai
crystalsemantics
dataforseobot
dataprovider
daum
deepcrawl
diffbot
domcopbot
dotbot
duckassistbot
duckduckbot
ev-crawler
exabot
experiancrawluk
facebookbot
genai
gptbot
go-http-client
google-extended
gptbot
grapeshot
httrack
img2dataset
imagesiftbot
lcc
linespider
ltx71 - (http://ltx71.com/)
magellan
magpie-crawler
mail.ru_bot
mauibot
megaindex
meta-externalagent
metajobbot
mj12bot
neevabot
netpeakcheckerbot
oai-searchbot
omgili
omgilibot
owler
panscient.com
perplexitybot
petalbot
piplbot
panscient.com
proximic
rainbot
riddler
rogerbot
scrapy
screaming frog seo spider
seekportbot
semanticbot
semanticscholarbot
semrushbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot-si
semrushbot-swa
sentibot
serpstatbot
seokicks
siteauditbot
sitecheckerbotcrawler
splitsignalbot
stormcrawler
the knowledge ai
timpibot
trendictionbot
velenpublicwebcrawler
wpbot
webprosbo
wellknownbot
wrtnbot
xovibot
yak
yaosoubot
yepbot
yeti
yisouspider
youbot
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- `ser-agent` is not a known field.
Comments