dllil.com
robots.txt

Robots Exclusion Standard data for dllil.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	dllil.com
Base Domain	dllil.com
Scan Status	Ok
Last Scan	5/22/2025, 5:09:05 AM
Next Scan	5/29/2025, 5:09:05 AM

Last Scan

Scanned	5/22/2025, 5:09:05 AM
URL	https://dllil.com/robots.txt
Domain IPs	104.21.88.135, 172.67.180.20, 2606:4700:3034::ac43:b414, 2606:4700:3036::6815:5887
Response IP	172.67.180.20
Found	Yes
Hash	b36362ffbfaffd9f055c34bb3a55c51717d557124f6d7f882b5354c40a6a5cc9
SimHash	36574981c4c7

Groups

ai2bot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazon-kendra

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot

Rule	Path
Disallow	/

Rule

Path

Disallow

awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

brightbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

duckassistbot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

friendlycrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

google-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

iaskspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

icc-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

img2dataset

Rule	Path
Disallow	/

Rule

Path

Disallow

kangaroo bot

Rule	Path
Disallow	/

Rule

Path

Disallow

linerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

machinelearningforpeacebot

Rule	Path
Disallow	/

Rule

Path

Disallow

meltwater

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalagent

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalfetcher

Rule	Path
Disallow	/

Rule

Path

Disallow

nicecrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

oai-searchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

pangubot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexity-user

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

piplbot

Rule	Path
Disallow	/

Rule

Path

Disallow

qualifiedbot

Rule	Path
Disallow	/

Rule

Path

Disallow

scoop.it

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot-ocob

Rule	Path
Disallow	/

Rule

Path

Disallow

sidetrade indexer bot

Rule	Path
Disallow	/

Rule

Path

Disallow

timpibot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

webzio-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

youbot

Rule	Path
Disallow	/

Rule

Path

Disallow

Comments

Cloudflare Managed Robots.txt to block AI related bots.

dllil.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

ai2bot

amazonbot

amazon-kendra

anthropic-ai

applebot

applebot-extended

awariorssbot

awariosmartbot

brightbot

bytespider

chatgpt-user

claudebot

diffbot

duckassistbot

facebookbot

friendlycrawler

google-extended

gptbot

iaskspider/2.0

icc-crawler

img2dataset

kangaroo bot

linerbot

machinelearningforpeacebot

meltwater

meta-externalagent

meta-externalfetcher

nicecrawler

oai-searchbot

omgili

omgilibot

pangubot

perplexitybot

perplexity-user

petalbot

piplbot

qualifiedbot

scoop.it

seekr

semrushbot-ocob

sidetrade indexer bot

timpibot

velenpublicwebcrawler

webzio-extended

youbot

Comments

dllil.com
robots.txt