Log In Sign Up

davidlebovitz.com
robots.txt

Robots Exclusion Standard data for davidlebovitz.com

Resource Scan

Scan Details

Site Domain davidlebovitz.com
Base Domain davidlebovitz.com
Scan Status Ok
Last Scan2025-08-10T13:46:50+00:00
Next Scan 2025-08-17T13:46:50+00:00

Last Scan

Scanned2025-08-10T13:46:50+00:00
URL https://davidlebovitz.com/robots.txt
Domain IPs 104.21.13.12, 172.67.167.225, 2606:4700:3030::6815:d0c, 2606:4700:3033::ac43:a7e1
Response IP 172.67.167.225
Found Yes
Hash 4913cc8efcbfc0140a53ef457258f99ba03e192cc75360385242b006e89a4a8e
SimHash 2938d704b283

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300
Back to top

Comments

  • Disallow: /cgi-bin/
  • Disallow: /i/
  • Disallow: /images/
  • Disallow: /*.gif$
  • Disallow: /*.png$
  • Disallow: /*.jpg$
  • Disallow: /*?
Back to top