Log In Sign Up

olitham.com
robots.txt

Robots Exclusion Standard data for olitham.com

Resource Scan

Scan Details

Site Domain olitham.com
Base Domain olitham.com
Scan Status Ok
Last Scan2026-01-29T06:11:59+00:00
Next Scan 2026-02-05T06:11:59+00:00

Last Scan

Scanned2026-01-29T06:11:59+00:00
URL https://olitham.com/robots.txt
Redirect https://www.olitham.com/robots.txt
Redirect Domain www.olitham.com
Redirect Base olitham.com
Domain IPs 104.21.68.199, 172.67.198.43, 2606:4700:3032::6815:44c7, 2606:4700:3037::ac43:c62b
Redirect IPs 104.21.68.199, 172.67.198.43, 2606:4700:3032::6815:44c7, 2606:4700:3037::ac43:c62b
Response IP 104.21.68.199
Found Yes
Hash 957be4ceb30fed41e0a8149d9d079f1f868d498b3a54d3f82d6e3c510e42731a
SimHash 40dec35061e6

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

chatgpt-user
gptbot

Rule Path
Allow /

google-extended
googleother

Rule Path
Allow /

anthropic-ai
claude-web

Rule Path
Allow /

ccbot

Rule Path
Allow /

facebookbot
meta-externalagent

Rule Path
Allow /

applebot
applebot-extended

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

ai2bot

Rule Path
Allow /

diffbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

amazonbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

googlebot
googlebot-image
googlebot-mobile
googlebot-video

Rule Path
Allow /

bingbot
bingpreview
msnbot

Rule Path
Allow /

slurp
yahoo-slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

twitterbot
facebookexternalhit
linkedinbot
slackbot
discordbot
whatsapp
telegrambot

Rule Path
Allow /

ia_archiver
archive.org_bot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.kotipathi.lk/sitemap.xml
sitemap https://www.kotipathi.lk/sitemap_index.xml
Back to top

Comments

  • Universal Access - Welcome Everyone!
  • Host
  • OpenAI (ChatGPT, GPT-4)
  • Google AI (Bard, Gemini, Google-Extended)
  • Anthropic (Claude)
  • Common Crawl (Research & Archive)
  • Meta AI
  • Apple Intelligence
  • Perplexity AI
  • Cohere AI
  • AI2 (Allen Institute)
  • Diffbot
  • ByteDance (TikTok)
  • Amazon AI
  • Yandex AI
  • =====================================================
  • Search Engine Bots - Priority Access
  • =====================================================
  • =====================================================
  • Social Media & Content Discovery
  • =====================================================
  • =====================================================
  • Archive & Research Bots
  • =====================================================
  • =====================================================
  • Performance & Monitoring
  • =====================================================
  • Cloudflare standards
  • Sitemap location (update with your actual sitemap URL)
  • =====================================================
  • Notes for Bot Operators:
  • =====================================================
  • - We support caching: Cache as needed
  • - We support indexing: Index freely
  • - We support AI training: Train responsibly
  • - Rate limiting: Be reasonable, we won't throttle ethical crawlers
  • - Contact: webmaster@kotipathi.lk
  • =====================================================
  • Last Updated: October 2025
  • =====================================================
Back to top

Warnings

  • `host` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top