docs.tigera.io
robots.txt

Robots Exclusion Standard data for docs.tigera.io

Resource Scan

Scan Details

Site Domain docs.tigera.io
Base Domain tigera.io
Scan Status Ok
Last Scan2024-08-28T19:53:13+00:00
Next Scan 2024-09-27T19:53:13+00:00

Last Scan

Scanned2024-08-28T19:53:13+00:00
URL https://docs.tigera.io/robots.txt
Domain IPs 13.251.96.10, 2406:da18:880:3800::c8, 2406:da18:880:3802::c8, 46.137.195.11
Response IP 52.74.166.77
Found Yes
Hash 564aedc66b3d1833be30db4ec1fb562624f4d332016ccb025ec59220e508e883
SimHash 531493eeed75

Groups

algolia crawler

Rule Path
Allow /calico/
Allow /calico-enterprise/
Allow /calico-cloud/

*

Rule Path
Disallow /archive/
Disallow /v3.14/
Disallow /v3.13/
Disallow /v3.12/
Disallow /v3.11/
Disallow /v3.10/
Disallow /v3.9/
Disallow /v3.8/
Disallow /v3.7/
Disallow /v3.6/
Disallow /v3.5/
Disallow /v3.4/
Disallow /v3.3/
Disallow /v3.2/
Disallow /v3.1/
Disallow /v3.0/
Disallow /v2.8/
Disallow /v2.7/
Disallow /v2.6/
Disallow /v2.5/
Disallow /v2.4/
Disallow /v2.3/
Disallow /v2.2/
Disallow /v2.1/
Disallow /calico/
Disallow /calico-enterprise/
Allow /calico/latest/
Allow /calico-enterprise/latest/

Comments

  • Allow Algolia crawler to crawl the Docusaurus-native sites only.
  • Disallow archive sites
  • Disallow all versioned paths
  • Allow latest versions