viaggi.corriere.it
robots.txt

Robots Exclusion Standard data for viaggi.corriere.it

Resource Scan

Scan Details

Site Domain viaggi.corriere.it
Base Domain corriere.it
Scan Status Ok
Last Scan2025-01-10T14:55:41+00:00
Next Scan 2025-01-17T14:55:41+00:00

Last Scan

Scanned2025-01-10T14:55:41+00:00
URL https://viaggi.corriere.it/robots.txt
Domain IPs 13.226.2.101, 13.226.2.52, 13.226.2.61, 13.226.2.65, 2600:9000:24bc:1600:13:a6de:4780:93a1, 2600:9000:24bc:2e00:13:a6de:4780:93a1, 2600:9000:24bc:3400:13:a6de:4780:93a1, 2600:9000:24bc:3800:13:a6de:4780:93a1, 2600:9000:24bc:9400:13:a6de:4780:93a1, 2600:9000:24bc:a800:13:a6de:4780:93a1, 2600:9000:24bc:ac00:13:a6de:4780:93a1, 2600:9000:24bc:ca00:13:a6de:4780:93a1
Response IP 3.164.206.42
Found Yes
Hash d292447aae13cd5747626e2b160d4d51a4ec378b302bc5478b25a4145d78f2e1
SimHash f90cda488717

Groups

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow */xmlrpc.php
Disallow */wp-*.php
Disallow */trackback/
Disallow */ricerca/tag/
Disallow */feed/*
Disallow /tag/luoghi/
Disallow *?doing_wp_cron=
Disallow */?app_v2=
Disallow /?s=
Disallow /itinerari-e-luoghi/cards/*/?%2F*
Disallow /weekend/cards/*/?%2F*
Disallow /eventi/cards/*/?%2F*
Disallow /news/cards/*/?%2F*
Disallow */?1
Disallow */?2
Disallow */?3
Disallow */?4
Disallow */?5
Disallow */?6
Disallow */?7
Disallow */?8
Disallow */?9
Disallow */?0
Disallow */?viewName=B1_FIRMA&

Other Records

Field Value
crawl-delay 5

Other Records

Field Value
sitemap https://viaggi.corriere.it/sitemap-news.xml
sitemap https://viaggi.corriere.it/sitemap.xml
sitemap https://viaggi.corriere.it/wp-content/uploads/sitemaps/sitemap-image.xml
sitemap https://viaggi.corriere.it/wp-content/uploads/sitemaps/sitemap-video.xml

Comments

  • Disallow: /gallery/
  • blocco concatenazione
  • end: test 21/5 su dataset 29000 / 37300 con img 26400
  • blocco param
  • end
  • Sitemap immagini gallery generata
  • Video sitemap generata