lorenagonzalezart.es
robots.txt

Robots Exclusion Standard data for lorenagonzalezart.es

Resource Scan

Scan Details

Site Domain lorenagonzalezart.es
Base Domain lorenagonzalezart.es
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-03-12T05:59:10+00:00
Next Scan 2024-06-10T05:59:10+00:00

Last Successful Scan

Scanned2021-09-21T12:05:36+00:00
URL https://lorenagonzalezart.es/robots.txt
Found Yes
Hash c5977d15072388110552b7678d41e937c91ff54cb620052f35995826725b7031
SimHash 6909eb5e5292

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /archives/
Disallow /comments/feed/
Disallow *?replytocom
Disallow /?s*
Disallow /*add-to-cart%3D*

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

Comments

  • disallow all files in these directories