Log In Sign Up

compliantlearningresources.com.au
robots.txt

Robots Exclusion Standard data for compliantlearningresources.com.au

Resource Scan

Scan Details

Site Domain compliantlearningresources.com.au
Base Domain compliantlearningresources.com.au
Scan Status Ok
Last Scan2025-09-11T22:27:33+00:00
Next Scan 2025-10-11T22:27:33+00:00

Last Scan

Scanned2025-09-11T22:27:33+00:00
URL https://compliantlearningresources.com.au/robots.txt
Domain IPs 104.26.4.122, 104.26.5.122, 172.67.74.223, 2606:4700:20::681a:47a, 2606:4700:20::681a:57a, 2606:4700:20::ac43:4adf
Response IP 172.67.74.223
Found Yes
Hash 049803648cf2c189c8e57ff21516ecf0c7c67e43ebfd1bb0862961b55df975a2
SimHash 0d229a1047b0

Groups

*

Rule Path
Allow /wp-content/uploads/2024/05/CLR-Favicon-1.ico
Disallow /wp-admin/
Disallow /wp-content/
Disallow /network/wp-content/uploads/*
Disallow /network/wp-content/blogs.dir/*
Disallow /?add-to-cart=*
Disallow /network/lotus/*
Disallow /rto-resources-2__trashed/*
Disallow /rto-resources-2/*
Disallow /code/*
Disallow /regenerateq/*
Disallow /network/sparkling-stars/files/*
Disallow /network/cascade-peak-school/files/*
Disallow /network/lotus/files/*
Disallow /network/awesome-landscapes/files/*
Disallow /network/biggerthanbig/files/*
Disallow /network/cascade-peak-performance/files/*
Disallow /network/sparkling-stars-eylc/files/*
Disallow /network/bounce-fitness/files/*
Disallow /network/cpschool/files/*
Disallow /network/cascade-peak-construction/files/*
Disallow /network/accountabilitynow/files/*
Disallow /network/lotus-v2/files/*
Disallow /network/cascade-peak-v2/files/*
Disallow /network/motionworks-allied-health-clinic/files/*

mj12bot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

google favicon

Rule Path
Allow /wp-content/uploads/2024/05/CLR-Favicon-1.ico

google-extended

Rule Path
Allow /

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

ccbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

metabot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://compliantlearningresources.com.au/sitemap_index.xml
Back to top

Comments

  • robots.txt for compliantlearningresources.com.au
  • Last updated: 2025-09-11
  • -----------------------
  • General Crawler Rules
  • -----------------------
  • -----------------------
  • SEO Audit Tools
  • -----------------------
  • -----------------------
  • Image Crawlers
  • -----------------------
  • -----------------------
  • AI and LLM Crawlers Access Policy
  • -----------------------
  • Allow Google Gemini (via Google-Extended)
  • Allow OpenAI's GPTBot
  • Allow Anthropic’s ClaudeBot
  • Allow Common Crawl (used by many LLMs)
  • Allow Perplexity AI's crawler
  • Allow Amazon’s AI crawler
  • Allow Meta's AI training crawler
  • -----------------------
  • XML Sitemap
  • -----------------------
Back to top