euroleghe.fantacalcio.it
robots.txt

Robots Exclusion Standard data for euroleghe.fantacalcio.it

Resource Scan

Scan Details

Site Domain euroleghe.fantacalcio.it
Base Domain fantacalcio.it
Scan Status Ok
Last Scan2024-05-07T00:14:52+00:00
Next Scan 2024-05-14T00:14:52+00:00

Last Scan

Scanned2024-05-07T00:14:52+00:00
URL https://euroleghe.fantacalcio.it/robots.txt
Domain IPs 108.156.133.106, 108.156.133.126, 108.156.133.34, 108.156.133.78, 2600:9000:2755:3400:3:57a7:e5c0:93a1, 2600:9000:2755:6600:3:57a7:e5c0:93a1, 2600:9000:2755:7400:3:57a7:e5c0:93a1, 2600:9000:2755:7800:3:57a7:e5c0:93a1, 2600:9000:2755:7e00:3:57a7:e5c0:93a1, 2600:9000:2755:8600:3:57a7:e5c0:93a1, 2600:9000:2755:aa00:3:57a7:e5c0:93a1, 2600:9000:2755:b400:3:57a7:e5c0:93a1
Response IP 108.156.133.78
Found Yes
Hash 9d6f6b16904cc075864e5161eb2c2368fe7519a3c73a759736d993f1b170eab3
SimHash 433cdd317652

Groups

ia_archiver

Rule Path
Disallow /

googlebot
adsbot-google
googlebot-image
msnbot
slurp

Rule Path
Disallow
Disallow /*?
Disallow /*/
Allow /*/info
Allow /*/squadre
Allow /*/home
Allow /*/rose
Allow /*/formazioni
Allow /*/calendario
Allow /*/classifica
Allow /*/statistiche
Allow /*/albo-oro
Allow /guide-leghe-fantacalcio/
Allow /iscrizione
Disallow /test

admantx-ussy04/3.2
admantx-sap/2.4
admantx-eusyncbatch01/3.1
admantx-adform/3.1
ias-ir/3.1
ias-va/3.1
ias-or/3.1
ias-sg/3.1
ias-jp/3.1
ias_crawler
ttd-contenuto
ahrefsbot
cloudfind
dotbot
blexbot
mj12bot
proximic

Rule Path
Disallow /

amazonadbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

criteobot/0.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

gumgum-bot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600
crawl-delay 3600
crawl-delay 3600

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Warnings

  • `user agent` is not a known field.