wolfgordon.com
robots.txt

Robots Exclusion Standard data for wolfgordon.com

Resource Scan

Scan Details

Site Domain wolfgordon.com
Base Domain wolfgordon.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-02-07T08:30:25+00:00
Next Scan 2026-05-08T08:30:25+00:00

Last Successful Scan

Scanned2025-09-16T04:47:44+00:00
URL https://wolfgordon.com/robots.txt
Redirect https://www.wolfgordon.com/robots.txt
Redirect Domain www.wolfgordon.com
Redirect Base wolfgordon.com
Domain IPs 172.66.41.32, 172.66.42.224, 2606:4700:3108::ac42:2920, 2606:4700:3108::ac42:2ae0
Redirect IPs 172.66.41.32, 172.66.42.224, 2606:4700:3108::ac42:2920, 2606:4700:3108::ac42:2ae0
Response IP 172.66.41.32
Found Yes
Hash 2d30eab489df21f887297c8892a300b358521595cfd715950ac6311c3f4b4451
SimHash 61341d2a073e

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/
Disallow /boards/
Disallow /admin/
Disallow /actions/

Other Records

Field Value
sitemap https://www.wolfgordon.com/sitemaps-1-sitemap.xml

Comments

  • robots.txt for https://www.wolfgordon.com/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow ChatGPT bot, as there's no benefit to allowing it to index your site
  • User-agent: GPTBot
  • Disallow: /
  • Disallow Google Bard and Vertex AI bots, as there's no benefit to allowing it to index your site
  • User-agent: Google-Extended
  • Disallow: /