scielo.cl
robots.txt

Robots Exclusion Standard data for scielo.cl

Resource Scan

Scan Details

Site Domain scielo.cl
Base Domain scielo.cl
Scan Status Ok
Last Scan2024-09-01T14:32:28+00:00
Next Scan 2024-10-01T14:32:28+00:00

Last Scan

Scanned2024-09-01T14:32:28+00:00
URL https://scielo.cl/robots.txt
Domain IPs 66.22.101.131
Response IP 66.22.101.131
Found Yes
Hash ad41282f51a2ab5de54a2c314d478c2918c7e460f0a25e63ddcae350f86712ef
SimHash 6549d391c695

Groups

mediapartners-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 11

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 12

adsbot-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 13

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 14

googlebot-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 16

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 17

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 18

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 19

*

Rule Path
Disallow /

Comments

  • Allow only major search spiders
  • Block all other spiders