Log In Sign Up

tonyherman.com
robots.txt

Robots Exclusion Standard data for tonyherman.com

Resource Scan

Scan Details

Site Domain tonyherman.com
Base Domain tonyherman.com
Scan Status Ok
Last Scan2025-08-29T07:02:50+00:00
Next Scan 2025-09-05T07:02:50+00:00

Last Scan

Scanned2025-08-29T07:02:50+00:00
URL https://tonyherman.com/robots.txt
Redirect https://www.tonyherman.com/robots.txt
Redirect Domain www.tonyherman.com
Redirect Base tonyherman.com
Domain IPs 192.0.79.156, 192.0.79.180
Redirect IPs 192.0.79.156, 192.0.79.180
Response IP 192.0.79.180
Found Yes
Hash 11ed3fb9aa9e524014305d366e19f40a65fac5b14f4954550577b9db460d3bfb
SimHash 13b65f103357

Groups

mediapartners-google*

Rule Path
Disallow /images/
Disallow /plesk-stat/
Disallow /author*

Other Records

Field Value
crawl-delay 10

serpstatbot

Rule Path
Disallow /

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

chatgpt-user

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

google-extended

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • 1/15/2023 TCH: This bot was bringing down the server:
  • AI Bots Crawl Delay
Back to top