Log In Sign Up

onetimesecret.com
robots.txt

Robots Exclusion Standard data for onetimesecret.com

Resource Scan

Scan Details

Site Domain onetimesecret.com
Base Domain onetimesecret.com
Scan Status Ok
Last Scan2025-11-14T00:06:43+00:00
Next Scan 2025-12-14T00:06:43+00:00

Last Scan

Scanned2025-11-14T00:06:43+00:00
URL https://onetimesecret.com/robots.txt
Domain IPs 138.199.24.219, 2400:52e0:1500::1274:1
Response IP 138.199.46.65
Found Yes
Hash 01c023f5bc553cfeb37f2512fda96b18417e33a9914938382fcf9bbffefc729d
SimHash 651e5a50a463

Groups

*

Rule Path
Allow /about
Allow /pricing
Allow /info/
Allow /translations
Allow /
Disallow /private/
Disallow /secret/
Disallow /receipt/
Disallow /account/
Disallow /recent/
Disallow /forgot/
Disallow /domains/
Disallow /signin
Disallow /signup
Disallow /logout
Disallow /incoming

Other Records

Field Value
crawl-delay 1

puppeteer

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /

claude-web

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://onetimesecret.com/sitemap-index.xml
Back to top

Comments

  • robots.txt for onetimesecret.com
  • Allow specific public pages first so they take precedence
  • Disallow sensitive or private areas
  • Crawl delay recommendation for high-traffic bots
  • Allow Puppeteer and LLM fetch requests
  • Sitemap declaration
Back to top