Log In Sign Up

curl.ly
robots.txt

Robots Exclusion Standard data for curl.ly

Resource Scan

Scan Details

Site Domain curl.ly
Base Domain curl.ly
Scan Status Ok
Last Scan2026-03-21T21:36:42+00:00
Next Scan 2026-04-04T21:36:42+00:00

Last Scan

Scanned2026-03-21T21:36:42+00:00
URL https://curl.ly/robots.txt
Domain IPs 104.21.81.250, 172.67.192.102, 2606:4700:3034::6815:51fa, 2606:4700:3036::ac43:c066
Response IP 172.67.192.102
Found Yes
Hash faf18a8c81e5ce4282c98f998cc62487cdaf1b960bf4f5420a8594ffcf29c1ef
SimHash 3f8a5e41e4f6

Groups

*

Rule Path
Allow /
Allow /features
Allow /features/
Allow /pricing
Allow /about
Allow /contact
Allow /use-cases
Allow /resources
Allow /privacy
Allow /terms
Allow /cookies
Allow /data-rights
Allow /p/
Disallow /app/
Disallow /admin/
Disallow /r/
Disallow /login
Disallow /signup
Disallow /forgot-password
Disallow /reset-password
Disallow /accept-invite
Disallow /auth-callback
Disallow /*?utm_*
Disallow /*?ref=*
Disallow /api/
Disallow /private/
Disallow /temp/

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

slackbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://curl.ly/sitemap.xml
sitemap https://ldxlhzrfreebrospyvnw.supabase.co/functions/v1/sitemap
Back to top

Comments

  • curl.ly robots.txt - SEO Optimized
  • Last updated: 2026-01-15
  • Default rules for all bots
  • Protected areas - disallow indexing
  • Googlebot - Full access to allowed pages
  • Bingbot
  • Social media bots - allow for previews
  • Sitemaps - All sitemaps for comprehensive indexing
  • Host directive
Back to top

Warnings

  • `host` is not a known field.
Back to top