Log In Sign Up

appunti.blog.kataweb.it
robots.txt

Robots Exclusion Standard data for appunti.blog.kataweb.it

Resource Scan

Scan Details

Site Domain appunti.blog.kataweb.it
Base Domain kataweb.it
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-11-02T17:24:15+00:00
Next Scan 2026-01-31T17:24:15+00:00

Last Successful Scan

Scanned2025-03-15T12:26:04+00:00
URL http://appunti.blog.kataweb.it/robots.txt
Domain IPs 34.242.108.154, 54.171.30.254
Response IP 54.171.30.254
Found Yes
Hash 241742869b725d286846b125ba778683527b30800a976044ed1c39c6fd746d73
SimHash 30044150c9a7

Groups

*

Rule Path
Allow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /

youbot

Rule Path
Disallow /

gumgum bot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

exabot

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

nabot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

lexxebot/1.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

wotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

fetch

Rule Path
Disallow /

nutch

Rule Path
Disallow /

umbot-ln

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

slurp

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

kbcrawl

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

quora-bot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

wesee:search

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

openbot

Rule Path
Disallow /

verticalsearch

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

npbot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

nnetseer crawler

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

dloader(naverrobot)

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

seoengbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /
Back to top