Log In Sign Up

beep.lk
robots.txt

Robots Exclusion Standard data for beep.lk

Resource Scan

Scan Details

Site Domain beep.lk
Base Domain beep.lk
Scan Status Ok
Last Scan2025-12-21T21:14:30+00:00
Next Scan 2026-01-20T21:14:30+00:00

Last Scan

Scanned2025-12-21T21:14:30+00:00
URL https://beep.lk/robots.txt
Domain IPs 18.155.68.34, 18.155.68.57, 18.155.68.85, 18.155.68.90
Response IP 18.155.68.85
Found Yes
Hash 6b6b83c986084a9271a4752a47404eb72ba30337974b45d2cb64203f7cb445dd
SimHash 6d579851e556

Groups

*

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

bingbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://beep.lk/sitemap.xml
Back to top

Comments

  • Robots.txt for Beep.lk
  • https://beep.lk/robots.txt
  • Allow all search engines and AI chatbots
  • Allow ChatGPT Search Bot (OpenAI)
  • Allow Bing Bot (Powers ChatGPT, Copilot, and others)
  • Allow Google Bot
  • Block GPTBot (AI training) - allows search but blocks training
  • Remove this if you want OpenAI to use your content for training
  • Allow Claude Bot (Anthropic)
  • Allow Perplexity Bot
  • Sitemaps
  • Crawl-delay (optional, helps prevent server overload)
  • Disallow admin or private areas (add if needed in future)
  • Disallow: /admin/
  • Disallow: /private/
Back to top