Log In Sign Up

davekb.com
robots.txt

Robots Exclusion Standard data for davekb.com

Resource Scan

Scan Details

Site Domain davekb.com
Base Domain davekb.com
Scan Status Ok
Last Scan2026-02-27T06:37:02+00:00
Next Scan 2026-03-06T06:37:02+00:00

Last Scan

Scanned2026-02-27T06:37:02+00:00
URL https://davekb.com/robots.txt
Domain IPs 172.105.104.97
Response IP 172.105.104.97
Found Yes
Hash 3ceeabc625e2376ed41a33e272922f7a354f500db9204b09a2f1bd56b3ce0d66
SimHash 203c4ca7cf51

Groups

*

Rule Path
Disallow /index.php
Disallow /random.php
Disallow /random_
Disallow /img.php*
Disallow /images

minibot(naverrobot)
googlebot-images

Rule Path
Disallow /

gptbot
google-bard
ccbot
anthropic-ai

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • Dosn't disallow in robots.txt now we use rel=nofollow
  • Disallow: /search.php
  • Disallow: /search.php*
  • Disallow: /keywords.php -- now done in the .php file
  • Images, etc bots
  • AI Bots
Back to top