iltuocomparatore.com
robots.txt

Robots Exclusion Standard data for iltuocomparatore.com

Resource Scan

Scan Details

Site Domain iltuocomparatore.com
Base Domain iltuocomparatore.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-26T13:02:18+00:00
Next Scan 2026-02-25T13:02:18+00:00

Last Successful Scan

Scanned2025-12-28T08:04:58+00:00
URL https://iltuocomparatore.com/robots.txt
Redirect https://www.iltuocomparatore.com/robots.txt
Redirect Domain www.iltuocomparatore.com
Redirect Base iltuocomparatore.com
Domain IPs 104.21.76.237, 172.67.202.23, 2606:4700:3030::ac43:ca17, 2606:4700:3036::6815:4ced
Redirect IPs 104.21.76.237, 172.67.202.23, 2606:4700:3030::ac43:ca17, 2606:4700:3036::6815:4ced
Response IP 104.21.76.237
Found Yes
Hash 6deea892184fd13ac6492a43464bcf467fb3ab5b7838b59bace099f520077d07
SimHash 6798485bd5c5

Groups

mediapartners-google

Rule Path
Allow /
Disallow /admin/
Disallow /search2.php
Disallow /jump.php
Disallow /contatti/form6.php
Disallow /privacy.php
Disallow /cookie_policy.php
Disallow /newsletter.php
Disallow /search.php
Disallow /pt/
Disallow /libri/
Disallow /top_categories.php
Disallow /fsearch.php
Disallow /merchants-mappa.php
Disallow /searchmerchants.php
Disallow /bsearch.php
Disallow /fsearch-moda.php
Disallow /fsearch-moda-brand.php
Disallow /category/
Disallow /categories.php
Disallow /brand/
Disallow /shoppinglist.php
Disallow /review/
Disallow /jump*.php

Other Records

Field Value
crawl-delay 1

etaospider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleagent-mariner

Rule Path
Disallow /

googleagent-shopping

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

devin

Rule Path
Disallow /

omgili

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

gemini-deep-research

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

mistralai-user

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

sidetrade indexer bot

Rule Path
Disallow /

google-notebooklm

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

copilot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

npbot-1/2.0

Rule Path
Disallow /

npbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.iltuocomparatore.com/sitemap.xml
sitemap https://www.iltuocomparatore.com/sitemap.php