k21academy.com
robots.txt

Robots Exclusion Standard data for k21academy.com

Resource Scan

Scan Details

Site Domain k21academy.com
Base Domain k21academy.com
Scan Status Ok
Last Scan2025-09-07T08:57:02+00:00
Next Scan 2025-10-07T08:57:02+00:00

Last Scan

Scanned2025-09-07T08:57:02+00:00
URL https://k21academy.com/robots.txt
Domain IPs 192.124.249.161
Response IP 192.124.249.161
Found Yes
Hash 4eece59d22cb53556fc79771d1ab0fb79ab3adfeb16f470315e66daa59abce4c
SimHash 192859128c33

Groups

oai-searchbot
chatgpt-user
perplexitybot
firecrawlagent
andibot
exabot
phindbot
youbot
claudebot
duckassistbot
apple-extended
baiduspider
yandex

Rule Path
Allow /

facebookexternalhit
linkedinbot
twitterbot

Rule Path
Allow /

*

Rule Path
Allow /azure-cloud-job-free-class/
Disallow /terms-and-conditions/
Disallow /privacy-policy/
Disallow /?preview=*
Disallow /course_breakdown/
Disallow /cb_category/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow *waitlist
Disallow /test-devops/
Disallow /testimonials/kamala-narayan-aws-solutions-architect-associate/
Disallow /amazon-web-services/aws-devops/aws-certified-devops-engineer-professional-dop-c01/
Disallow /aws-job-program/
Disallow /job-oriented-azure-cloud-program/
Disallow /docker-kubernetes-job-oriented-program/
Disallow /devsecops-training/
Disallow /job-oriented-gcp-cloud-program/
Disallow /oracle-cloud-job-program/
Disallow /azure-data-cloud-job-program/
Disallow /awsdva-job-program/
Disallow /terraform-infrastructure-automation-certification-training/
Disallow /google-cloud-program/

Other Records

Field Value
crawl-delay 4

Other Records

Field Value
sitemap https://k21academy.com/sitemap.xml

Comments

  • Allow OpenAI & other AI/search-related crawlers
  • Explicitly allow major social scrapers
  • General rule for all other bots
  • Explicitly allow the Azure Cloud Free Class page
  • Disallow sensitive or irrelevant areas
  • Disallow waitlist & promo URLs
  • Disallow duplicate or job program listing pages