cilentonotizie.it
robots.txt

Robots Exclusion Standard data for cilentonotizie.it

Resource Scan

Scan Details

Site Domain cilentonotizie.it
Base Domain cilentonotizie.it
Scan Status Ok
Last Scan2024-11-15T08:35:14+00:00
Next Scan 2024-11-22T08:35:14+00:00

Last Scan

Scanned2024-11-15T08:35:14+00:00
URL https://cilentonotizie.it/robots.txt
Redirect https://www.cilentonotizie.it/robots.txt
Redirect Domain www.cilentonotizie.it
Redirect Base cilentonotizie.it
Domain IPs 31.11.35.125
Redirect IPs 31.11.35.125
Response IP 31.11.35.125
Found Yes
Hash 844636e92ae7985b329b1d99a0172baab92e0202a371be5f4ba499a6e8441438
SimHash 2a844380c5a5

Groups

*

Rule Path
Disallow /download/
Disallow /utility/
Disallow /chisiamo/

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

bingbot

Rule Path
Disallow

bingpreview

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

msnbot-media

Rule Path
Disallow

msnbot

Rule Path
Disallow

archive.org_bot

Rule Path
Disallow

naverbot

Rule Path
Disallow

pinterestbot

Rule Path
Disallow

screaming frog seo spider

Rule Path
Disallow

seznambot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

telegrambot

Rule Path
Disallow

twitterbot

Rule Path
Disallow

yeti

Rule Path
Disallow

quora-bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

viberbot

Rule Path
Disallow /

yak

Rule Path
Disallow /

yandex

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot

Rule Path
Disallow /

discordbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

adidxbot

Rule Path
Disallow

zoombot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /