Log In Sign Up

thedave.me
robots.txt

Robots Exclusion Standard data for thedave.me

Resource Scan

Scan Details

Site Domain thedave.me
Base Domain thedave.me
Scan Status Ok
Last Scan2026-01-20T19:53:31+00:00
Next Scan 2026-02-03T19:53:31+00:00

Last Scan

Scanned2026-01-20T19:53:31+00:00
URL https://thedave.me/robots.txt
Domain IPs 104.21.70.247, 172.67.141.2, 2606:4700:3031::6815:46f7, 2606:4700:3037::ac43:8d02
Response IP 172.67.141.2
Found Yes
Hash 3cd6a3c5636f1da304f06a97b91280df3b892d9c8cb03ab40cc687e1bceef98e
SimHash aa850d856770

Groups

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://thedave.me/author-sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Disallow: /
Back to top