Log In Sign Up

usesparrow.com
robots.txt

Robots Exclusion Standard data for usesparrow.com

Resource Scan

Scan Details

Site Domain usesparrow.com
Base Domain usesparrow.com
Scan Status Ok
Last Scan2026-02-06T17:07:10+00:00
Next Scan 2026-02-20T17:07:10+00:00

Last Scan

Scanned2026-02-06T17:07:10+00:00
URL https://usesparrow.com/robots.txt
Domain IPs 3.171.198.38, 3.171.198.47, 3.171.198.49, 3.171.198.5
Response IP 3.171.198.5
Found Yes
Hash 0dfdcc10ebd9aacffd8db769841224e6e0d067f70ca4edec83afe4f4ba6c7234
SimHash 6c0ce824ef80

Groups

*

Rule Path
Allow /
Disallow /admin
Disallow /api/
Disallow /file/
Disallow /.well-known/
Disallow /local_assets/
Disallow /brands-app/
Disallow /*.json$
Disallow /*.xml$
Disallow /*.config$
Allow /sitemap.xml
Allow /favicon.ico

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://usesparrow.com/sitemap.xml
Back to top

Comments

  • Robots.txt for https://usesparrow.com
  • Generated: 2024
  • Allow all crawlers by default
  • Disallow admin and internal paths
  • Block specific file types from indexing
  • Allow specific important files
  • Search engine specific rules
  • Google
  • Bing
  • Common AI/ML crawlers - you may want to block these
  • Sitemap location
  • Default crawl delay for other bots
Back to top