andrewlock.net
robots.txt

Robots Exclusion Standard data for andrewlock.net

Resource Scan

Scan Details

Site Domain andrewlock.net
Base Domain andrewlock.net
Scan Status Ok
Last Scan2025-03-25T20:20:58+00:00
Next Scan 2025-04-08T20:20:58+00:00

Last Scan

Scanned2025-03-25T20:20:58+00:00
URL https://andrewlock.net/robots.txt
Domain IPs 104.21.1.30, 172.67.151.230, 2606:4700:3035::ac43:97e6, 2606:4700:3036::6815:11e
Response IP 172.67.151.230
Found Yes
Hash 20269b0f832c3d09f0072c7a1420a7c8fac82c1f3cde1ca4b05915e5d2e0e35a
SimHash 60088b61c113

Groups

gptbot

Rule Path
Disallow /
Allow /about/

chatgpt-user

Rule Path
Disallow /
Allow /about/

google-extended

Rule Path
Disallow /
Allow /about/

ccbot

Rule Path
Disallow /
Allow /about/

perplexitybot

Rule Path
Disallow /
Allow /about/

facebookbot

Rule Path
Disallow /
Allow /about/

omgilibot

Rule Path
Disallow /
Allow /about/

anthropic-ai

Rule Path
Disallow /
Allow /about/

cohere-ai

Rule Path
Disallow /
Allow /about/

*

Rule Path
Disallow /404
Disallow /404.html
Disallow /error
Disallow /error.html
Disallow /offline
Disallow /offline.html

Other Records

Field Value
sitemap https://andrewlock.net/sitemap.xml