cartemigliori.it
robots.txt
Robots Exclusion Standard data for cartemigliori.it
Resource Scan
Scan Details
| Site Domain | cartemigliori.it |
| Base Domain | cartemigliori.it |
| Scan Status | Ok |
| Last Scan | 2024-11-11T20:03:25+00:00 |
| Next Scan | 2024-12-11T20:03:25+00:00 |
Last Scan
| Scanned | 2024-11-11T20:03:25+00:00 |
| URL | https://cartemigliori.it/robots.txt |
| Domain IPs | 104.21.72.93, 172.67.179.198, 2606:4700:3034::ac43:b3c6, 2606:4700:3036::6815:485d |
| Response IP | 104.21.72.93 |
| Found | Yes |
| Hash | 89d429c90e3b4e7e90d0186d01687212158864b76fce20f742347e79a131448a |
| SimHash | d26973738c4b |
Groups
zoombot
ahrefsbot
aihitbot
alphaseobot
alphaseobot-sa
backlinkcrawler
baiduspider
baiduspider-image
baiduspider-video
blackwidow
blexbot
bubing
careerbot
ccbot
chinaclaw
cliqzbot
cloudservermarketspider
coccoc
custo
disco
domain re-animator bot
dotbot
dotbot
download\ demon
ecatch
eccp/1.0 (search@eniro.com)
eirgrabber
emailsiphon
emailwolf
exabot
express\ webpictures
extractorpro
eyenetie
ezooms robot
flashget
fr-crawler
getright
getweb!
gigabot
go!zilla
go-ahead-got-it
grabnet
grafula
grapeshot
grapeshotcrawler/2.0
gsa-crawler (enterprise; t4-knhh62cdkc2w3; gsa_manage@nikon-sys.co.jp)
gsa-crawler
haosouspider
hmview
httrack
ia_archiver
icc-crawler/2.0
iccrawler
image\ stripper
image\ sucker
implisensebot
indy\ library
interget
internet\ ninja
jetcar
jobboersebot
jobs.de-robot
joc\ web\ spider
kraken
larbin
leechftp
linkdexbot/2.1
linkstats
lipperhey-kaus-australis
magpie-crawler
mass\ downloader
meanpathbot
megaindex.com
megaindex.ru
megaindex.ru
megaindex.ru/2.0
metajobbot
midown\ tool
mindupbot
mister\ pix
navroad
nearsite
nerdybot
net\ vampire
netants
netestate ne crawler
netestate ne crawler (+http://www.website-datenbank.de/)
netspider
netzip
obot
octopus
offline\ explorer
offline\ navigator
openhosebot
pagegrabber
papa\ foto
pavuk
pcbrowser
perl lwp
pi-monster
pimonster
pimonster
plista
plukkie
proximic
qwantify
r6_commentreader
realdownload
reget
rogerbot
rogerbot
safednsbot
screaming frog seo spider
searchmetricsbot
semrushbot
seodiver
seokicks-robot
seoscanners.net
seznambot
sg-orbiter
sistrix
sistrix
sistrix crawler
sitesnagger
smartdownload
sogou spider
sogou spider
spbot
spiderbot
superbot
superhttp
surfbot
surveybot
takeout
teleport\ pro
thumbsniper
trendictionbot
turnitin robot
turnitinbot
um-ic
unisterbot
uptimerobot/2.0
voideye
wbsearchbot
web\ image\ collector
web\ sucker
webauto
webcopier
webfetch
webgo\ is
webleacher
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
wget
widow
wiseguys robot
wotbox
wwwoffle
xaldon\ webspider
yadirectfetcher
yandex
yandexmobilebot
youdaobot
zeus
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /varie/ |
| Disallow | /includes/ |
| Disallow | /contenuti/ |
| Disallow | /layout/ |
| Disallow | /landing/ |
| Allow | /*.js* |
| Allow | /*.css* |
| Allow | /*.js?* |
| Allow | /*.css?* |
| Allow | /*.xml?* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.cartemigliori.it/sitemap.xml |
Warnings
- 2 invalid lines.