cataloniatoday.cat
robots.txt
Robots Exclusion Standard data for cataloniatoday.cat
Resource Scan
Scan Details
| Site Domain | cataloniatoday.cat |
| Base Domain | cataloniatoday.cat |
| Scan Status | Ok |
| Last Scan | 2024-11-06T05:19:08+00:00 |
| Next Scan | 2024-11-13T05:19:08+00:00 |
Last Scan
| Scanned | 2024-11-06T05:19:08+00:00 |
| URL | https://cataloniatoday.cat/robots.txt |
| Redirect | https://www.cataloniatoday.cat/robots.txt |
| Redirect Domain | www.cataloniatoday.cat |
| Redirect Base | cataloniatoday.cat |
| Domain IPs | 52.138.153.35 |
| Redirect IPs | 52.138.153.35 |
| Response IP | 52.138.153.35 |
| Found | Yes |
| Hash | 556e2bf73af0c2edd45b9faba428bade9d5f5b3a9cbf17bba0dbf19672b21cf4 |
| SimHash | 4228554837ba |
Groups
aihitbot
alphaseobot
alphaseobot-sa
blexbot
careerbot
cliqzbot
megaindex
owlin
owlin bot
owlin bot v. 3.0
queryseekerspider
seokicks-robot
semrush
semrushbot
sogou
tkbot
vagabondo
wbsearchbot
dotbot
grapeshot
icjobs
linkdexbot
magpie-crawler
proximic
scrapy
scrapybot
sentibot
adidxbot
ahrefsbot
blp_bbot
blp_bbot/0.1
baiduspider
baiduspider+
baiduspider-image+
bingpreview
exabot/3.0
mj12bot
naverbot
semrushbot
snapbot/1.0
yandex
yandexbot
yeti
youdaobot
dtsearchspider
ichiro
magpie-crawler
moget
sogou spider
screaming frog seo spider
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /administrator/ |
| Disallow | /cache/ |
| Disallow | /components/ |
| Disallow | /images/ |
| Disallow | /includes/ |
| Disallow | /installation/ |
| Disallow | /language/ |
| Disallow | /libraries/ |
| Disallow | /media2/ |
| Disallow | /modules/ |
| Disallow | /plugins/ |
| Disallow | /templates2/ |
| Disallow | /tmp/ |
| Disallow | /xmlrpc/ |
| Disallow | /cgi-bin/ |
| Disallow | /serveirestringit.html |
| Disallow | /*.raw |
| Disallow | /edicio/ |
| Disallow | /*/*.raw |
| Disallow | /*/*.raw?* |
| Disallow | *.jpg?ESBORRAM* |
Comments