Log In Sign Up

cybercoders.com
robots.txt

Robots Exclusion Standard data for cybercoders.com

Resource Scan

Scan Details

Site Domain cybercoders.com
Base Domain cybercoders.com
Scan Status Ok
Last Scan2025-08-24T06:52:46+00:00
Next Scan 2025-09-23T06:52:46+00:00

Last Scan

Scanned2025-08-24T06:52:46+00:00
URL https://cybercoders.com/robots.txt
Domain IPs 192.103.251.130
Response IP 192.103.251.130
Found Yes
Hash 5ad36fce0707e7e0b8359eeeaedf6bf950446a77e8f1aecb3b98a0584e6bfec6
SimHash 44189540e4c0

Groups

*

Rule Path
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin
Allow /

gptbot
chatgpt-user
chatgpt-user/2.0
oai-searchbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

claude-web
claudebot
anthropic-ai

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

google-extended

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

ccbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

perplexitybot
perplexity-user

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

bingbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

amazonbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

applebot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

bytespider

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

youbot
mistralai-user
duckassistbot
cohere-ai
ai2bot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin

facebookbot

Rule Path
Allow /
Allow /search
Allow /job-detail/
Allow /careeradvice/
Allow /aboutus
Allow /employer
Allow /contactus
Allow /sitemap.md
Disallow /apply
Disallow /register
Disallow /resume-collector
Disallow /profile
Disallow /member-onboarding
Disallow /account
Disallow /admin
Back to top

Other Records

Field Value
sitemap https://www.cybercoders.com/sitemap.xml
Back to top

Comments

  • Robots.txt for CyberCoders
  • Updated to include AI crawler directives and AI-friendly sitemap
  • Standard web crawlers (Google, Bing, etc.)
  • AI Crawlers - OpenAI
  • These crawlers will receive markdown content with job search instructions
  • Crawl-delay: 1
  • AI Crawlers - Anthropic (Claude)
  • Crawl-delay: 1
  • AI Crawlers - Google AI
  • Crawl-delay: 1
  • AI Crawlers - Common Crawl (used by many AI companies)
  • Crawl-delay: 1
  • AI Crawlers - Perplexity
  • Crawl-delay: 1
  • AI Crawlers - Microsoft
  • Crawl-delay: 1
  • AI Crawlers - Amazon
  • Crawl-delay: 1
  • AI Crawlers - Apple
  • Crawl-delay: 1
  • AI Crawlers - ByteDance
  • Crawl-delay: 1
  • AI Crawlers - Other AI Services
  • Crawl-delay: 1
  • AI Crawlers - Meta AI (Note: FacebookBot should be in social media crawlers)
  • Crawl-delay: 1
  • Instructions for AI: Start with https://www.cybercoders.com/sitemap.md for complete guidance
Back to top

Warnings

  • `ai sitemap` is not a known field.
Back to top