Log In Sign Up

sf.net
robots.txt

Robots Exclusion Standard data for sf.net

Resource Scan

Scan Details

Site Domain sf.net
Base Domain sf.net
Scan Status Ok
Last Scan2024-05-17T03:24:14+00:00
Next Scan 2024-05-24T03:24:14+00:00

Last Scan

Scanned2024-05-17T03:24:14+00:00
URL https://sf.net/robots.txt
Redirect https://sourceforge.net/robots.txt
Redirect Domain sourceforge.net
Redirect Base sourceforge.net
Domain IPs 104.18.20.237, 104.18.21.237
Redirect IPs 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, 2606:4700::6812:d95
Response IP 104.18.12.149
Found Yes
Hash 21b32f18ea8a1cf3e38af9f575391bd6f86b8d37b6cbb142e1b803367e03e96f
SimHash 7980388fc7e7

Groups

gptbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

*

Rule Path
Allow /p/*/ci/master/tree/$
Allow /p/*/ci/main/tree/$
Allow /p/*/ci/default/tree/$
Allow /p/*/HEAD/tree/$
Disallow /p/*/code/
Disallow /p/*/git/
Disallow /p/*/svn/
Disallow /p/*/hg/
Disallow /p/*/search
Disallow /p/*/code-0/
Disallow /p/*/tree/
Disallow /p/*/search_feed/
Disallow /p/*/search_help/
Disallow /p/*/wiki/*/edit
Disallow /p/*/wiki/browse_tags/
Disallow /p/*/discussion/create_topic/
Disallow /p/*/discussion/stats
Disallow /*/ci/
Disallow /*/commit_browser
Disallow /*/commit_browser_data
Disallow /*?barediff=
Disallow /*?diff=
Disallow /*/bugs/new/
Disallow /*/support-requests/new/
Disallow /*/patches/new/
Disallow /*/feature-requests/new/
Disallow /*/tickets/new/
Disallow /*/attachment/
Disallow /projects/*/files/*/stats/timeline
Disallow /projects/*/files/stats/timeline
Disallow /projects/*/files/*/stats/map
Disallow /projects/*/files/stats/map
Disallow /projects/*/files/*/stats/os
Disallow /projects/*/files/stats/os
Disallow /projects/*/files/stats/json
Disallow /projects/*/files/*/stats/json
Disallow /projects/*/files/latest/download
Disallow /projects/*/latest.json
Disallow /projects/*/report_inappropriate
Disallow /projects/*/reviews/new
Disallow /projects/*/rss
Disallow /projects/*/lists/*/unsubscribe
Disallow /projects/*/moderate_review
Disallow /projects/*/rate_review
Disallow /projects/*/best_release.html
Disallow /projects/*/postdownload
Disallow /projects/*/get_updates
Disallow /*_escaped_fragment_
Disallow /auth/
Disallow /auth/do_login
Disallow /software/visit?
Disallow /software/compare/*-vs-*-vs-*-vs-*-vs-
Disallow /software/compare/*/add-software
Disallow /software/product/*/reviews/new
Disallow /software/product/*/claim
Disallow /software/vendors/new
Disallow /software/vendors/inquire
Disallow /software/*?q=
Disallow /software/*%26q%3D
Disallow /software/*?categories=
Disallow /software/*%26categories%3D
Disallow /software/*?company_sizes=
Disallow /software/*%26company_sizes%3D
Disallow /software/*?deployment=
Disallow /software/*%26deployment%3D
Disallow /software/*?free_options=
Disallow /software/*%26free_options%3D
Disallow /software/*?integrates_with=
Disallow /software/*%26integrates_with%3D
Disallow /software/*?org_types=
Disallow /software/*%26org_types%3D
Disallow /software/*?api=
Disallow /software/*%26api%3D
Disallow /software/*?regions=
Disallow /software/*%26regions%3D
Disallow /software/*?support=
Disallow /software/*%26support%3D
Disallow /software/*?training=
Disallow /software/*%26training%3D
Disallow /software/*?feature_
Disallow /software/*%26feature_
Disallow /rest/
Disallow /user/registration
Disallow */bin_counts
Disallow */milestone_counts
Disallow */stats_data
Disallow *//users$
Disallow */users$
Disallow /u/*/link/
Disallow /p/*/link/
Disallow */feed.rss
Disallow */feed.atom
Disallow /p/*/news/feed$
Disallow */activity/feed
Disallow */activity/pjax
Disallow /p/*/admin/
Disallow /directory/release_feed
Disallow *?css-reload=
Disallow *%26css-reload%3D
Disallow /settings/mirror_choices
Disallow /directory/tp
Disallow /software/tp
Disallow /sd-jobs
Disallow /directory/*?
Allow /directory/*?q
Disallow /directory/*?q*&
Allow /directory/*?page=
Disallow *?style=flat
Disallow *%26style%3Dflat
Disallow *?style=threaded
Disallow *%26style%3Dthreaded
Disallow *?viewmonth=
Disallow *%26viewmonth%3D
Disallow *?viewmont=
Disallow *%26viewmont%3D
Disallow *?viewday=
Disallow *%26viewday%3D

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://sourceforge.net/sitemap.xml
sitemap https://sourceforge.net/allura_sitemap/sitemap.xml
sitemap https://sourceforge.net/directory_sitemap.xml
sitemap https://sourceforge.net/software_sitemap.xml
sitemap https://sourceforge.net/blog/sitemap_index.xml
sitemap https://sourceforge.net/articles/sitemap_index.xml
Back to top

Comments

  • /directory param rules: permit ?q= and ?page= on their own, but no other params
  • longest match takes precedence. listed in length order
  • mailman flags
Back to top