Log In Sign Up

video.espresso.repubblica.it
robots.txt

Robots Exclusion Standard data for video.espresso.repubblica.it

Resource Scan

Scan Details

Site Domain video.espresso.repubblica.it
Base Domain repubblica.it
Scan Status Ok
Last Scan2025-02-21T05:51:22+00:00
Next Scan 2025-03-23T05:51:22+00:00

Last Scan

Scanned2025-02-21T05:51:22+00:00
URL https://video.espresso.repubblica.it/robots.txt
Domain IPs 18.161.111.3, 18.161.111.30, 18.161.111.80, 18.161.111.99
Response IP 18.165.140.21
Found Yes
Hash 3e05bb02d61bbcc6ad0b781a8a4d1fa6e120490839ef0ebdcbb1a2736f049f84
SimHash 380441508987

Groups

*

Rule Path
Allow /static/images/
Disallow /static/
Disallow /ssi/
Disallow /php/

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

gumgum bot

Rule Path
Disallow /

peer39_crawler

Rule Path
Disallow /

web-archive-net.com.bot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

exabot

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

nabot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

lexxebot/1.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

wotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

fetch

Rule Path
Disallow /

nutch

Rule Path
Disallow /

umbot-ln

Rule Path
Disallow /

europarchive.org

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

slurp

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

kbcrawl

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

quora-bot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

primalbot

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

peer39_crawler/1.0

Rule Path
Disallow /

wesee:search

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

openbot

Rule Path
Disallow /

verticalsearch

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

npbot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

nnetseer crawler

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

dloader(naverrobot)

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

seoengbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

linkarchiver

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

pixray-seeker

Rule Path
Disallow /
Back to top