davidayo.com
robots.txt

Robots Exclusion Standard data for davidayo.com

Resource Scan

Scan Details

Site Domain davidayo.com
Base Domain davidayo.com
Scan Status Ok
Last Scan2024-10-04T22:34:05+00:00
Next Scan 2024-10-11T22:34:05+00:00

Last Scan

Scanned2024-10-04T22:34:05+00:00
URL https://davidayo.com/robots.txt
Redirect https://www.davidayo.com/robots.txt
Redirect Domain www.davidayo.com
Redirect Base davidayo.com
Domain IPs 172.66.41.19, 172.66.42.237, 2606:4700:3108::ac42:2913, 2606:4700:3108::ac42:2aed
Redirect IPs 172.66.41.19, 172.66.42.237, 2606:4700:3108::ac42:2913, 2606:4700:3108::ac42:2aed
Response IP 172.66.42.237
Found Yes
Hash 8904860bdeed8a8df268e8cd36da0d646e0c107c9f37a34a4a94a88f94561bb0
SimHash f9149c46c7c2

Groups

*

Rule Path
Disallow
Disallow /admin/
Disallow /private/
Disallow /health/
Disallow /legal/
Disallow /insurance/
Allow /public/

Other Records

Field Value
sitemap https://www.davidayo.com/sitemap.xml
sitemap https://www.davidayo.com/blog-sitemap.xml

Comments

  • we use ExpressJS as our technology platform
  • This robots.txt allows indexing of all site paths
  • Allow all crawlers