Log In Sign Up

conduit.link
robots.txt

Robots Exclusion Standard data for conduit.link

Resource Scan

Scan Details

Site Domain conduit.link
Base Domain conduit.link
Scan Status Ok
Last Scan2025-11-17T04:39:35+00:00
Next Scan 2025-12-17T04:39:35+00:00

Last Scan

Scanned2025-11-17T04:39:35+00:00
URL https://conduit.link/robots.txt
Domain IPs 104.21.39.187, 172.67.148.66, 2606:4700:3031::ac43:9442, 2606:4700:3036::6815:27bb
Response IP 172.67.148.66
Found Yes
Hash 9ef5875d8d3511cf5946fb36708a297b86749d7492afe2d0151694f93eec6a7a
SimHash 4b2cce80ede2

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /api/
Disallow /_next/
Disallow /dashboard/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

slackbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://conduit.link/sitemap.xml
Back to top

Comments

  • Robots.txt for Conduit Link
  • https://conduit.link
  • Allow all crawlers
  • Sitemap location
  • Crawl delay for respectful crawling
  • Disallow admin and private areas (if any exist)
  • Allow specific important bots faster access
  • Block bad bots
  • Allow social media bots for better sharing
Back to top