kentrota.com
robots.txt

Robots Exclusion Standard data for kentrota.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	kentrota.com
Base Domain	kentrota.com
Scan Status	Ok
Last Scan	2025-04-24T01:26:26+00:00
Next Scan	2025-05-08T01:26:26+00:00

Last Scan

Scanned	2025-04-24T01:26:26+00:00
URL	https://kentrota.com/robots.txt
Domain IPs	172.66.40.111, 172.66.43.145, 2606:4700:3108::ac42:286f, 2606:4700:3108::ac42:2b91
Response IP	172.66.40.111
Found	Yes
Hash	b36362ffbfaffd9f055c34bb3a55c51717d557124f6d7f882b5354c40a6a5cc9
SimHash	36574981c4c7

Groups

ai2bot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazon-kendra

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot

Rule	Path
Disallow	/

Rule

Path

Disallow

awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

brightbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

duckassistbot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

friendlycrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

google-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

iaskspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

icc-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

img2dataset

Rule	Path
Disallow	/

Rule

Path

Disallow

kangaroo bot

Rule	Path
Disallow	/

Rule

Path

Disallow

linerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

machinelearningforpeacebot

Rule	Path
Disallow	/

Rule

Path

Disallow

meltwater

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalagent

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalfetcher

Rule	Path
Disallow	/

Rule

Path

Disallow

nicecrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

oai-searchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

pangubot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexity-user

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

piplbot

Rule	Path
Disallow	/

Rule

Path

Disallow

qualifiedbot

Rule	Path
Disallow	/

Rule

Path

Disallow

scoop.it

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot-ocob

Rule	Path
Disallow	/

Rule

Path

Disallow

sidetrade indexer bot

Rule	Path
Disallow	/

Rule

Path

Disallow

timpibot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

webzio-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

youbot

Rule	Path
Disallow	/

Rule

Path

Disallow

Comments

Cloudflare Managed Robots.txt to block AI related bots.

kentrota.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

ai2bot

amazonbot

amazon-kendra

anthropic-ai

applebot

applebot-extended

awariorssbot

awariosmartbot

brightbot

bytespider

chatgpt-user

claudebot

diffbot

duckassistbot

facebookbot

friendlycrawler

google-extended

gptbot

iaskspider/2.0

icc-crawler

img2dataset

kangaroo bot

linerbot

machinelearningforpeacebot

meltwater

meta-externalagent

meta-externalfetcher

nicecrawler

oai-searchbot

omgili

omgilibot

pangubot

perplexitybot

perplexity-user

petalbot

piplbot

qualifiedbot

scoop.it

seekr

semrushbot-ocob

sidetrade indexer bot

timpibot

velenpublicwebcrawler

webzio-extended

youbot

Comments

kentrota.com
robots.txt