motleyhealth.com
robots.txt

Robots Exclusion Standard data for motleyhealth.com

Resource Scan

Scan Details

Site Domain motleyhealth.com
Base Domain motleyhealth.com
Scan Status Ok
Last Scan2024-11-11T07:00:37+00:00
Next Scan 2024-11-18T07:00:37+00:00

Last Scan

Scanned2024-11-11T07:00:37+00:00
URL https://www.motleyhealth.com/robots.txt
Domain IPs 104.21.5.89, 172.67.133.59, 2606:4700:3032::6815:559, 2606:4700:3032::ac43:853b
Response IP 104.21.5.89
Found Yes
Hash cdd9c2fb98a1cf784c6f1c78748c9bdc65b4d58dbb000787f280ad5645214719
SimHash c1151e151622

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow *?replytocom
Disallow /downloads/
Disallow /cgi-bin
Disallow /wp-admin
Disallow /wp-content
Disallow /wp-includes
Disallow /wget/
Disallow /httpd/
Disallow /i/
Disallow /f/
Disallow /t/
Disallow /c/
Disallow /j/
Disallow /pt/cgi-bin
Disallow /pt/wp-admin
Disallow /pt/wp-includes
Disallow /pt/wp-content
Disallow /pt/wget/
Disallow /pt/httpd/
Disallow /pt/i/
Disallow /pt/f/
Disallow /pt/t/
Disallow /pt/c/
Disallow /pt/j/

*

Rule Path
Allow /wp-content/plugins/wp-minify/

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

googlebot

Rule Path
Allow /