Log In Sign Up

highprogrammer.com
robots.txt

Robots Exclusion Standard data for highprogrammer.com

Resource Scan

Scan Details

Site Domain highprogrammer.com
Base Domain highprogrammer.com
Scan Status Ok
Last Scan2024-09-03T10:43:41+00:00
Next Scan 2024-10-03T10:43:41+00:00

Last Scan

Scanned2024-09-03T10:43:41+00:00
URL http://highprogrammer.com/robots.txt
Redirect http://www.highprogrammer.com/robots.txt
Redirect Domain www.highprogrammer.com
Redirect Base highprogrammer.com
Domain IPs 2001:470:0:208::403e:8c08, 64.62.140.8
Redirect IPs 2001:470:0:208::403e:8c08, 64.62.140.8
Response IP 64.62.140.8
Found Yes
Hash 97158319bf470f229d402f0b626984af9031469fd5dec93f5a6dc4e411b8c08f
SimHash 8a9813304ded

Groups

*

Rule Path
Disallow /cgi-bin/quoter
Disallow /cgi-bin/debug

npbot

Rule Path
Disallow /

naverrobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /
Back to top

Comments

  • Name Protect's bot. They have a stupid bot crawling the
  • web looking for infringment. That's nice, I suppose, but
  • not on my nickle, thanks.
  • No idea who these monkeys are, but they specifically crawled the entirety
  • quoter against the above request (even after reading robots.txt). Worse,
  • they crawled all of quoter in about two minutes, slamming me. Idiots.
  • Hopefully they'll respect this. Ought to check back later, see if I can
  • track them down and send them hate mail.
  • SEO and/or marketing robots
  • Mysterious and incredibly rude
Back to top