Log In Sign Up

candidateportal.creativecircle.com
robots.txt

Robots Exclusion Standard data for candidateportal.creativecircle.com

Resource Scan

Scan Details

Site Domain candidateportal.creativecircle.com
Base Domain creativecircle.com
Scan Status Ok
Last Scan2025-09-16T15:22:12+00:00
Next Scan 2025-10-16T15:22:12+00:00

Last Scan

Scanned2025-09-16T15:22:12+00:00
URL https://candidateportal.creativecircle.com/robots.txt
Domain IPs 192.103.251.130
Response IP 192.103.251.130
Found Yes
Hash a3ec498c8881f1cbd00f1176a07d5b8847a401f7274b2eea325a17bbbceb0752
SimHash 44189540e440

Groups

*

Rule Path
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin
Allow /

gptbot
chatgpt-user
chatgpt-user/2.0
oai-searchbot
openai-searchbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

claude-web
claude-user
claudebot
anthropic-ai

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

google-extended

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

ccbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

perplexitybot
perplexity-user

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

bingbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

amazonbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

applebot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

bytespider

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

youbot
mistralai-user
duckassistbot
cohere-ai
ai2bot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

facebookbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin
Back to top

Other Records

Field Value
sitemap https://www.cybercoders.com/sitemap.xml
Back to top

Comments

  • Robots.txt for CyberCoders
  • Updated to include AI crawler directives and AI-friendly sitemap
  • Standard web crawlers (Google, Bing, etc.)
  • AI Crawlers - OpenAI
  • These crawlers will receive markdown content with job search instructions
  • Crawl-delay: 1
  • AI Crawlers - Anthropic (Claude)
  • Crawl-delay: 1
  • AI Crawlers - Google AI
  • Crawl-delay: 1
  • AI Crawlers - Common Crawl (used by many AI companies)
  • Crawl-delay: 1
  • AI Crawlers - Perplexity
  • Crawl-delay: 1
  • AI Crawlers - Microsoft
  • Crawl-delay: 1
  • AI Crawlers - Amazon
  • Crawl-delay: 1
  • AI Crawlers - Apple
  • Crawl-delay: 1
  • AI Crawlers - ByteDance
  • Crawl-delay: 1
  • AI Crawlers - Other AI Services
  • Crawl-delay: 1
  • AI Crawlers - Meta AI (Note: FacebookBot should be in social media crawlers)
  • Crawl-delay: 1
  • Instructions for AI: Start with https://www.cybercoders.com/sitemap.md for complete guidance
Back to top

Warnings

  • `ai sitemap` is not a known field.
Back to top