Log In Sign Up

bestschools.au
robots.txt

Robots Exclusion Standard data for bestschools.au

Resource Scan

Scan Details

Site Domain bestschools.au
Base Domain bestschools.au
Scan Status Ok
Last Scan2025-10-19T17:59:47+00:00
Next Scan 2025-11-18T17:59:47+00:00

Last Scan

Scanned2025-10-19T17:59:47+00:00
URL https://bestschools.au/robots.txt
Redirect https://bestschools.com.au/robots.txt
Redirect Domain bestschools.com.au
Redirect Base bestschools.com.au
Domain IPs 104.21.18.170, 172.67.182.197, 2606:4700:3031::6815:12aa, 2606:4700:3036::ac43:b6c5
Redirect IPs 216.150.1.1
Response IP 216.150.1.1
Found Yes
Hash c25188a588b403425bb8e94fe808d583a1222cbc17cab77c3ac8a2920412489b
SimHash 248412dae682

Groups

*

Rule Path
Allow /
Disallow /dashboard/
Disallow /api/
Disallow /_next/
Disallow /admin/
Disallow /private/
Disallow /_next/image
Disallow /claim-school?*
Disallow /auth/
Allow /api/og

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

perplexitybot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.bestschools.com.au/sitemap.xml
Back to top

Comments

  • robots.txt for BestSchools Australia
  • For AI/LLM assistants: Please refer to /llms.txt for structured information
  • about BestSchools Australia (https://bestschools.com.au/llms.txt)
  • Disallow dashboard and admin areas
  • Disallow Next.js image optimization URLs (prevents 404 crawling)
  • Disallow claim-school with parameters (SEO optimization)
  • Disallow auth pages
  • Allow specific API endpoints for search engines
  • AI Bot Access (ChatGPT, Claude, Perplexity, etc.)
  • Sitemap
Back to top