Log In Sign Up

newsforge.com
robots.txt

Robots Exclusion Standard data for newsforge.com

Resource Scan

Scan Details

Site Domain newsforge.com
Base Domain newsforge.com
Scan Status Ok
Last Scan2024-11-18T17:45:27+00:00
Next Scan 2024-11-25T17:45:27+00:00

Last Scan

Scanned2024-11-18T17:45:27+00:00
URL http://newsforge.com/robots.txt
Redirect https://sourceforge.net/robots.txt
Redirect Domain sourceforge.net
Redirect Base sourceforge.net
Domain IPs 208.80.122.154, 208.80.122.51, 208.80.123.110, 208.80.123.233, 2600:1803:10a::2c, 2600:1803:10b::6b, 2600:1803:10c::d, 2600:1803:10d::7a
Redirect IPs 104.18.12.149, 104.18.13.149, 2606:4700::6812:c95, 2606:4700::6812:d95
Response IP 104.18.12.149
Found Yes
Hash 23659b02a231b7ea70844d44e714dce736c5de8bfd8433b5088a30838a90e994
SimHash 789474cfc7e5

Groups

*

Rule Path
Allow /p/*/ci/master/tree/$
Allow /p/*/ci/main/tree/$
Allow /p/*/ci/default/tree/$
Allow /p/*/HEAD/tree/$
Disallow /p/*/code/
Disallow /p/*/git/
Disallow /p/*/svn/
Disallow /p/*/hg/
Disallow /p/*/search
Disallow /p/*/code-0/
Disallow /p/*/tree/
Disallow /p/*/search_feed/
Disallow /p/*/search_help/
Disallow /p/*/wiki/*/edit
Disallow /p/*/wiki/browse_tags/
Disallow /p/*/discussion/create_topic/
Disallow /p/*/discussion/stats
Disallow /*/ci/
Disallow /*/commit_browser
Disallow /*/commit_browser_data
Disallow /*?barediff=
Disallow /*?diff=
Disallow /*/tarball
Disallow /code-snapshots/
Disallow /*/bugs/new/
Disallow /*/support-requests/new/
Disallow /*/patches/new/
Disallow /*/feature-requests/new/
Disallow /*/tickets/new/
Disallow /*/edit/
Disallow /*/attachment/
Disallow /*/_discuss/
Disallow /projects/*/files/*/stats/timeline
Disallow /projects/*/files/stats/timeline
Disallow /projects/*/files/*/stats/map
Disallow /projects/*/files/stats/map
Disallow /projects/*/files/*/stats/os
Disallow /projects/*/files/stats/os
Disallow /projects/*/files/stats/json
Disallow /projects/*/files/*/stats/json
Disallow /projects/*/files/latest/download
Disallow /projects/*/latest.json
Disallow /projects/*/report_inappropriate
Disallow /projects/*/reviews/new
Disallow /projects/*/rss
Disallow /projects/*/lists/*/unsubscribe
Disallow /projects/*/moderate_review
Disallow /projects/*/rate_review
Disallow /projects/*/best_release.html
Disallow /projects/*/postdownload
Disallow /projects/*/get_updates
Disallow /*_escaped_fragment_
Disallow /auth/
Disallow /auth/do_login
Disallow /software/link?
Disallow /software/link$
Disallow /software/visit?
Disallow /software/visit$
Disallow /software/compare/*-vs-*-vs-*-vs-
Disallow /software/compare/*/add-software
Disallow /software/product/*/reviews/new
Disallow /software/product/*/claim
Disallow /software/vendors/new
Disallow /software/vendors/inquire
Disallow /software/*?q=
Disallow /software/*%26q%3D
Disallow /software/*?categories=
Disallow /software/*%26categories%3D
Disallow /software/*?company_sizes=
Disallow /software/*%26company_sizes%3D
Disallow /software/*?deployment=
Disallow /software/*%26deployment%3D
Disallow /software/*?free_options=
Disallow /software/*%26free_options%3D
Disallow /software/*?integrates_with=
Disallow /software/*%26integrates_with%3D
Disallow /software/*?org_types=
Disallow /software/*%26org_types%3D
Disallow /software/*?api=
Disallow /software/*%26api%3D
Disallow /software/*?regions=
Disallow /software/*%26regions%3D
Disallow /software/*?support=
Disallow /software/*%26support%3D
Disallow /software/*?training=
Disallow /software/*%26training%3D
Disallow /software/*?feature_
Disallow /software/*%26feature_
Disallow /rest/
Allow /rest/*/activity
Disallow /user/registration
Disallow */bin_counts
Disallow */milestone_counts
Disallow */stats_data
Disallow *//users$
Disallow */users$
Disallow /u/*/link/
Disallow /p/*/link/
Disallow /p/*/donate
Disallow */feed.rss
Disallow */feed.atom
Disallow /p/*/news/feed
Disallow /arris/*/feed$
Disallow */activity/feed
Disallow */activity/pjax
Disallow /p/*/admin/
Disallow /directory/release_feed
Disallow *?css-reload=
Disallow *%26css-reload%3D
Disallow /settings/mirror_choices
Disallow /directory/os%3Awindows/?q=
Disallow /directory/tp
Disallow /software/tp
Disallow /sd-jobs
Disallow /about/mirror-agreement
Disallow /t7
Disallow /p/*/wiki/*/history
Disallow /p/*/*/milestone/
Disallow /p/*/activity/
Disallow /u/*/wiki/*
Disallow /u/*/activity/
Allow /u/*/profile/$
Disallow /u/*/*/
Disallow /*/wp-login.php
Disallow /blog/tag/
Disallow /blog/category/
Disallow /articles/tag/
Disallow /articles/category/open-source/
Disallow /articles/category/developer-tools/
Disallow /articles/category/artificial-intelligence/
Disallow /articles/category/database/
Disallow /articles/category/mobile/
Disallow /articles/category/payroll/
Disallow /articles/category/telephony/
Disallow /articles/category/networking/
Disallow /articles/category/cloud/
Disallow /articles/category/general/
Disallow /articles/category/software/
Disallow /p/allura/chat/
Disallow /p/allura/tickets/
Disallow /about/maxnav
Disallow /directory/*?
Allow /directory/*?q
Disallow /directory/*?q*&
Allow /directory/*?page=
Allow /directory/*license%3D
Allow /directory/*natlanguage%3D
Allow /directory/*developmentstatus%3D
Disallow *?style=flat
Disallow *%26style%3Dflat
Disallow *?style=threaded
Disallow *%26style%3Dthreaded
Disallow *?viewmonth=
Disallow *%26viewmonth%3D
Disallow *?viewmont=
Disallow *%26viewmont%3D
Disallow *?viewday=
Disallow *%26viewday%3D
Back to top

Other Records

Field Value
sitemap https://sourceforge.net/sitemap.xml
sitemap https://sourceforge.net/allura_sitemap/sitemap.xml
sitemap https://sourceforge.net/directory_sitemap.xml
sitemap https://sourceforge.net/software_sitemap.xml
sitemap https://sourceforge.net/blog/sitemap_index.xml
sitemap https://sourceforge.net/articles/sitemap_index.xml
Back to top

Comments

  • /directory param rules: permit ?q= and ?page= on their own, but no other params
  • longest match takes precedence. listed in length order
  • temp - allow google to revisit these pages and get the noindex:
  • mailman flags
Back to top