criteria.yml

# This YAML file provides information about each criterion.
# The top level hash key is the identifier of the criterion,
# with the following keys inside:
#   category: MUST|SHOULD|SUGGESTED # required
#   future: true|false # optional, default false
#   na_allowed: true|false # optional, default false
#   met_url_required: true|false # optional, default false
#   description: ...HTML description... # required
#   details: ...HTML description... # optional
#   met_placeholder: ...special met_placeholder text... # optional
#   unmet_placeholder: ...special unmet_placeholder text... # optional
#   na_placeholder: ...special na_placeholder text... # optional
#   met_suppress: true|false # optional, default false. Suppress justification?
#   unmet_suppress: true|false # optional, default false. Suppress just.
#   autofill: ...text describing *ideas* for how to autofill, in markdown
#
# Please note that autofill text is *not* a guarantee of how it will
# be implemented (or even if it will be automated); it is simply a way
# to record ideas that might be implemented.  Also, autofill is often
# used to *guess* correct values, to help speed filling in the form;
# we only force values if we are confident in their answer or are willing
# to require that the information be provided in a specific way.
# We've focused on GitHub, but where reasonable we'd like to create
# portability layers and support other forges like SourceForge,
# Savannah, Bitbucket, and GitLab; we'd also like to be able to support
# standalone sites (e.g., detect use of Bugzilla and work with that).
#
# We use an *ordered* map (omap), because the order below is used to determine
# the order of presentation.
--- !!omap
# BASICS
#
# Basic Project Website Content
- description_good:
    category: MUST
    description: >
      The project website MUST succinctly describe what the software does
      (what problem does it solve?).
    details: >
      This MUST be in language that potential users can understand
      (e.g., it uses minimal jargon).
    met_placeholder: >
      (Optional) Where is the succinct description?  Consider providing a URL.
    autofill: >
      We could try examining the project website or README for an
      early sentence of the form (This software|project name).
      It's unclear how effective this would be; we could survey a number
      of existing projects to see if there are common patterns.
- interact:
    category: MUST
    description: >
      The project website MUST provide information on how to:
      obtain, provide feedback (as bug reports or enhancements),
      and contribute to the software.
    met_placeholder: >
      (Optional) Where is this information?  Consider providing URL(s).
    autofill: >
      Look on the project website for words like "download|obtain",
      "contribute|bug report|feedback|enhancement", and
      "contribute|pull request|merge request".
- contribution:
    category: MUST
    met_url_required: true
    description: >
      The information on how to contribute MUST explain the
      contribution process (e.g., are pull requests used?)
    details: >
      We presume that
      <a href="https://guides.github.com/activities/contributing-to-open-source/">projects on GitHub use issues and pull requests</a>
      unless otherwise noted.
      This information can be short, e.g., stating that the project uses
      pull requests, an issue tracker, or posts to a mailing list (which one?)
    met_placeholder: >
      (URL required) What is the process? What URL explains it?
    autofill: >
      Look on the project website for words like
      "contribute|bug report|feedback|enhancement" and
      "contribute|pull request|merge request".
- contribution_requirements:
    category: SHOULD
    met_url_required: true
    description: >
      The information on how to contribute SHOULD include the
      requirements for acceptable contributions
      (e.g., a reference to any required coding standard).
    unmet_placeholder: >
      Why are the requirements so obvious that this information isn't needed?
    autofill: >
      Look for a CONTRIBUTING{,.md,.txt,.html} file.
# OSS License
- floss_license:
    category: MUST
    description: >
      The software MUST be released as FLOSS.
    details: >
      FLOSS is software released in a way that meets the
      <a href="https://opensource.org/osd-annotated">Open Source Definition</a>
      or
      <a href="http://www.gnu.org/philosophy/free-sw.en.html">Free Software
      Definition</a>.
      Examples of such licenses include the
      <a href="http://creativecommons.org/publicdomain/zero/1.0/">CC0</a>,
      <a href="https://opensource.org/licenses/MIT">MIT</a>, <a
      href="https://opensource.org/licenses/BSD-2-Clause">BSD 2-clause</a>,
      <a href="https://opensource.org/licenses/BSD-3-Clause">BSD
      3-clause revised</a>, <a
      href="https://opensource.org/licenses/Apache-2.0">Apache 2.0</a>,
      <a href="https://opensource.org/licenses/lgpl-license">Lesser
      GNU General Public License (LGPL)</a>, and the <a
      href="https://opensource.org/licenses/gpl-license">GNU General Public
      License (GPL)</a>.
      For our purposes, this means that the license MUST be:
      <ul>
      <li><a href="https://opensource.org/licenses">an approved license
      by the Open Source Initiative (OSI)</a>, or</li>
      <li><a href="https://www.gnu.org/licenses/license-list.html">a free
      license as approved by the Free Software Foundation (FSF)</a>, or</li>
      <li><a href="https://www.debian.org/legal/licenses/">a free license
      acceptable to Debian main</a>, or</li>
      <li><a
      href="https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing">a
      "good" license according to Fedora</a>.</li>
      </ul>
      The software MAY also be licensed other ways
      (e.g., "GPLv2 or proprietary" is acceptable).
      <br>
      <br>
      <em>Rationale</em>: These criteria are designed for FLOSS projects,
      so we need to ensure that they're only used where they apply.
      Some projects may be mistakenly considered FLOSS even though they
      are not (e.g., they might not have any license, in which case the
      defaults of the country's legal system apply, or they might use a
      non-FLOSS license).  Unusual licenses can cause long-term problems
      for FLOSS projects and are more difficult for tools to handle.
      We expect that more advanced badges would set a higher bar (e.g.,
      that it <em>must</em> be released under an OSI-approved license).
    autofill: >
      We currently use GitHub's API to request license information.
      This API examines the LICENSE file using the gem licensee.
      This only works on GitHub, and only in simple cases
      (when there's a single license).
      We could in addition look for the file LICENSE.spdx for the
      entry 'PackageLicenseDeclared' - these aren't included often,
      but it would let people declare *exactly* what license is in use.
      We could add running Ben Balter's gem
      [licensee](https:/benbalter/licensee), this is the same
      library used by GitHub but we can then apply it outside of GitHub.
      [Licensee issue #85](https:/benbalter/licensee/issues/85)
      proposes adding LICENSE.spdx support to licensee.
      See [&#8220;Open Source Licensing by the Numbers&#8221; by Ben Balter](https://speakerdeck.com/benbalter/open-source-licensing-by-the-numbers).
      We could also do more specialized analysis (e.g., looking for
      license information in various packaging formats) - this would
      probably be best done by improving some existing library like licensee.
- floss_license_osi:
    category: SUGGESTED
    description: >
      It is SUGGESTED that any required license(s) be
      <a href="https://opensource.org/licenses">approved by the
      Open Source Initiative (OSI).</a>
    details: >
      The OSI uses a rigorous approval process to determine
      which licenses are OSS.
    met_suppress: true
    autofill: >
      Currently we compare the license to see if it's on a simple list
      from OSI.  We could add support for automatically getting the list
      from elsewhere (e.g., OSI or SPDX), and support more complex
      license structures (OR, AND, and WITH).
- license_location:
    category: MUST
    met_url_required: true
    description: >
      The project MUST post license(s) in a standard location.
    details: >
      E.g., as a top-level file named LICENSE or COPYING.
      License filenames MAY be followed by an extension such as ".txt" or ".md".
    autofill: >
      Look for files with the name LICENSE, COPYING, or COPYING-(name)
      optionally extensions .txt or .md.  Name could include
      GPL, LGPL, and MIT.
# Documentation
- documentation_basics:
    category: MUST
    description: The project MUST provide basic documentation for the software.
    details: >
      This documentation must be in some media (such as text or video)
      that includes:
      how to install it, how to start it, how to use it (possibly with a
      tutorial using examples), and how to use it securely (e.g., what to do
      and what not to do) if that is an appropriate topic for the software.
      The security documentation need not be long.
    met_placeholder: >
      (Optional) What URL(s) are the starting points to for documentation?
    autofill: >
      Look for the project web page,
      or a direct links from it on the same site,
      that includes words like "install(ation)?", "use|using",
      and "security|secure".
- documentation_interface:
    category: MUST
    description: >
      The project MUST include reference documentation that describes
      its interface.
    details: >
      The project MAY use hypertext links to
      non-project material as documentation.
    autofill: >
      Look for a "docs" or "doc" directory, or the main site, with
      .txt, .html, .md, .tex extension and uses the word "interface".
# Other
- sites_https:
    category: MUST
    description: >
      The project sites (website, repository, and download URLs)
      MUST support HTTPS using TLS.
    details: >
      You can get free certificates from
      <a href="https://letsencrypt.org/">Let's Encrypt</a>.
    autofill: >
      Look at project, repo, and download URLs. https is okay, http is not.
      Typically anything supporting HTTPS also supports TLS, so
      it's probably not worth trying to detect that specifically.
- discussion:
    category: MUST
    description: >
      The project MUST have one or more mechanisms
      for discussion (including proposed changes and issues) that are
      searchable, allow messages and topics to be addressed by URL,
      enable new people to participate in some of the discussions, and
      do not require client-side installation of proprietary software.
    details: >
      Examples of acceptable mechanisms include
      GitHub issue and pull request discussions, Bugzilla, Mantis, and Trac.
      Asynchronous discussion mechanisms (like IRC) are acceptable if
      they meet these criteria; make sure there is a URL-addressable
      archiving mechanism.
      Proprietary Javascript, while discouraged, is permitted.
    autofill: >
      Currently if it's on GitHub we assume they will use its mechanisms.
      We could do the same for other forges like
      SourceForge, GitLab, and Savannah.
      We could also look for links from the project website that suggest
      the use of Bugzilla, Mantis, or Trac.
- english:
    category: SHOULD
    description: >
      The project SHOULD include documentation in English and be able
      to accept bug reports and comments about code in English.
    details: >
      English is currently the <a
      href="https://en.wikipedia.org/wiki/Lingua_franca">lingua franca</a>
      of computer technology; supporting English increases the number
      of different potential developers and reviewers worldwide.
      A project can meet this criterion even if its core developers'
      primary language is not English.
    autofill: >
      Look at project and/or repo page.  If it's in English,
      then clearly the project can accept English.
      We can use more general tools to detect the natural language,
      or simple mechanisms like using dictionaries/spellcheckers to
      see if it's mostly English.
#
# CHANGE CONTROL
#
# Public version-controlled source repository
- repo_public:
    category: MUST
    description: >
      The project MUST have a version-controlled source repository that
      is publicly readable and has a URL.
    details: >
      The URL MAY be the same as the project URL.
      The project MAY use private (non-public) branches in specific
      cases while the change is not publicly released (e.g., for fixing
      a vulnerability before it is revealed to the public).
    autofill: >
      We currently assume that being on GitHub is enough.
      Again, consider supporting other forges.
- repo_track:
    category: MUST
    description: >
      The source repository MUST track what changes were made, who made
      the changes, and when the changes were made.
    autofill: >
      If it uses git, subversion (svn), mercurial (hg), or even CVS
      this would normally be met.
- repo_interim:
    category: MUST
    description: >
      To enable collaborative review, the project's source repository MUST
      include interim versions for review between releases;
      it MUST NOT include only final releases.
    details: >
      Projects MAY choose to omit specific interim versions
      from their public source repositories
      (e.g., ones that fix specific non-public security vulnerabilities,
      may never be publicly released, or include material that cannot
      be legally posted and are not in the final release).
    autofill: >
      Consider checking if there are many versions.  If versions are
      only posted once every 3+ months, there's probably a problem.
- repo_distributed:
    category: SUGGESTED
    description: >
      It is SUGGESTED that common distributed version control software
      be used (e.g., git).
    details: >
      Git is not specifically required and projects
      can use centralized version control software
      (such as subversion) with justification.
    autofill: >
      Look for git, subversion (svn), and mercurial (hg).
# Unique version numbering
- version_unique:
    category: MUST
    description: >
      The project MUST have a unique version number for each release intended
      to be used by users.
    autofill: >
      Look for downloads with version numbers in the filename, or
      version tags in a git repo.
- version_semver:
    category: SUGGESTED
    description: >
      It is SUGGESTED that the
      <a href="http://semver.org">Semantic Versioning (SemVer) format</a>
      be used for releases.
    details: >
      Commit IDs (or similar) MAY be used as version numbers. They are
      unique, but note that these can cause problems for users as they
      may not be able to determine whether or not they're up-to-date.
    met_suppress: true
    autofill: >
      Look for git tags (at least on GitHub) that have version format, e.g.,
      v?[0-9]+\.[0-9]+\.[0-9]+.*   The prefixed 'v' is sometimes used in tags.
- version_tags:
    category: SUGGESTED
    description: >
      It is SUGGESTED that projects identify each release within their
      version control system.
      For example, it is SUGGESTED that those using git identify each release
      using git tags.
    autofill: >
      Again,
      look for git tags (at least on GitHub) that have version format, e.g.,
      v?[0-9]+\.[0-9]+\.[0-9]+.*   The prefixed 'v' is sometimes used in tags.
# Release notes (ChangeLog)
- release_notes:
    category: MUST
    met_url_required: true
    description: >
      The project MUST provide, in each release, release notes that are
      a human-readable <i>summary</i> of major changes in that release.
    details: >
      The release notes MUST NOT be the output of a version control log
      (e.g., the "git log" command results are not release notes).<br
      /> The release notes MAY be implemented in a variety of ways.
      Many projects provide them in a file named "NEWS", "CHANGELOG",
      or "ChangeLog", optionally with extensions such as ".txt", ".md",
      or ".html".  Historically the term "change log" meant a log of
      <i>every</i> change, but to meet these criteria what is needed
      is a human-readable summary.  The release notes MAY instead be
      provided by version control system mechanisms such as the <a
      href="https:/blog/1547-release-your-software">GitHub
      Releases workflow</a>.<br />
      <b>Rationale</b>:
      Release notes are important because they help users decide whether
      or not they will want to update, and what the impact would be (e.g.,
      if the new release fixes vulnerabilities).
    autofill: >
      Look for version-controlled files named NEWS, CHANGELOG, or ChangeLog
      (optionally with an extension .txt, .md, or .html).
      Also check to see if it uses a GitHub Releases workflow.
- release_notes_vulns:
    category: MUST
    description: >
      The release notes MUST identify every publicly known vulnerability
      that is fixed in each new release.
    autofill: >
      Examine the release notes (per above) to see if they include
      CVE identifiers.  Also,
      use a vulnerability database such as the
      National Vulnerability Database (NVD) to try to identify
      publicly known vulnerabilities in this software, and see if all of
      them are mentioned in the release notes.
#
# REPORTING
#
# Bug-reporting process
- report_process:
    category: MUST
    description: >
      The project MUST provide a process for users to submit bug reports
      (e.g., using an issue tracker or a mailing list).
    met_url_required: true
    autofill: >
      On GitHub we presume that at least issue trackers can be used.
      Search CONTRIBUTING (if it exists) for a phrase like "bug reports".
- report_tracker:
    category: SHOULD
    description: >
      The project SHOULD use an issue tracker for tracking individual issues.
    unmet_placeholder: Why is there no issue tracker?
    autofill: >
      On GitHub we presume that at least issue trackers can be used.
      Search CONTRIBUTING (if it exists) for a phrase like "issue tracker".
- report_responses:
    category: MUST
    description: >
      The project MUST acknowledge a majority of bug reports submitted in
      the last 2-12 months (inclusive); the response need not include a fix.
    autofill: >
      Examine issue tracker, and see what is marked as a "bug".
      Exclude the top contributor(s) as reported by the repo changes.
      Then examine the responses for the rest.
      Measuring "top contributors" is tricky; one potential rule is
      those who contribute more than 10% of the system, or, when you
      sort people by their contributions, the ones who cumulatively wrote
      2/3s of the system.  However, these measures may be difficult to obtain
      in a short time - perhaps just exclude anyone listed in AUTHORS or
      CREDITS, or someone listed in the last X commits?
- enhancement_responses:
    category: SHOULD
    description: >
      The project SHOULD respond to most enhancement requests in the last
      2-12 months (inclusive). The project MAY choose not to respond.
    autofill: >
      Examine issue tracker, and see what is marked as an "enhancement".
      Exclude the top contributor(s) as reported by the repo changes.
      Then examine the responses for the rest.
- report_archive:
    category: MUST
    met_url_required: true
    description: >
      The project MUST have a publicly available archive for reports and
      responses for later searching.
    autofill: >
      If on GitHub, Savannah, SourceForge, GitLab, this is probably fine.
# Vulnerability report process
- vulnerability_report_process:
    category: MUST
    met_url_required: true
    description: >
      The project MUST publish the process for reporting vulnerabilities
      on the project site.
    details: >
      E.g., a clearly designated mailing address on
      https://PROJECTSITE/security,
      often in the form security@example.org.
      This MAY be the same as its bug reporting process.
    autofill: >
      Look for phrase like "vulnerability reporting" or
      "how to report vulnerabilities" in various documents
      like README, CONTRIBUTING, or a doc/* file.
- vulnerability_report_private:
    category: MUST
    na_allowed: true
    met_url_required: true
    description: >
      If private vulnerability reports are supported, the project MUST
      include how to send the information in a way that is kept private.
    details: >
      E.g., a private defect report submitted on the web using TLS
      or an email encrypted using OpenPGP.
      If private vulnerability reports are not supported this criterion is
      automatically met.
    autofill: >
      Look for phrase like "private vulnerability reporting" or
      "how to report private vulnerabilities" in various documents
      like README, CONTRIBUTING, or a doc/* file, and *also*
      look for an OpenPGP key to use for encrypting the information.
      Sadly, while Bugzilla easily supports this,
      GitHub doesn't currently support this in its issue tracker, see
      https:/isaacs/github/issues/37
- vulnerability_report_response:
    category: MUST
    description: >
      The project's initial response time for any vulnerability report received
      in the last 6 months MUST be less than or equal to 14 days.
    autofill: >
      If GitHub is used, and the issue tracker has something marked
      "security" or "vulnerability", measure the appropriate times.
      GitHub doesn't support private reports, though, so a lot of people
      won't use this.
      If Bugzilla is used, grab the information publicly available and
      track this.
#
# QUALITY
#
# Working build system
- build:
    category: MUST
    na_allowed: true
    description: >
      If the software requires building for use,
      the project MUST provide a working build system that can
      automatically rebuild the software from source code.
    details: >
      A build system determines what actions need to occur to rebuild the
      software (and in what order), and then performs those steps.
      <b>Rationale</b>: If a project needs to be built but there is no
      working build system, then potential co-developers will not be
      able to easily contribute and many security analysis tools will be
      ineffective. Criteria for a working build system are not applicable
      if there is no need to build anything for use.
    autofill: >
      See build_common_tools.
- build_common_tools:
    category: SUGGESTED
    na_allowed: true
    description: >
      It is SUGGESTED that common tools be used for building the software.
    details: >
      For example, Maven, Ant, cmake, the autotools, make, or rake.
    autofill: >
      Look for files that suggest the use of a common build systems, e.g.:
      the autotools (configure.ac, makefile.am),
      traditional make (Makefile, makefile),
      cmake, rake, ant, maven, etc.
      See http://www.dwheeler.com/essays/releasing-floss-software.html
- build_floss_tools:
    category: SHOULD
    na_allowed: true
    description: The project SHOULD be buildable using only FLOSS tools.
    autofill: >
      If it's packaged in a Linux distribution repo that is FLOSS-only,
      then this is met.  That includes official Debian distribution
      (but not contrib or non-free) or Fedora's official distribution.
      If it's only source code file extensions that don't normally require
      building (.py, .rb, etc.), then it is likely NA.
      If the build_common_tools answers are themselves FLOSS, then this
      is more likely to *also* be true.  The only *real* way to check this
      is to install and build on a system that only has FLOSS, and we
      won't have the time to do that ourselves.
# Automated test suite
- test:
    category: MUST
    description: >
      The project MUST have at least one automated test suite that is publicly
      released as FLOSS (this test suite may be maintained as a
      separate FLOSS project).
    details: >
      The project MAY have multiple automated test suites (e.g., one
      that runs quickly, vs. another that is more thorough but requires
      special equipment).
      <b>Rationale</b>: Automated test suites immediately help detect a
      variety of problems.  A large test suite can find more problems, but
      even a small test suite can detect problems and provide a framework
      to build on.
    autofill: >
      See test_invocation, and use its value here also.
- test_invocation:
    category: SHOULD
    description: >
      A test suite SHOULD be invocable in a standard way for that language.
    details: >
      For example, "make check", "mvn test", or "rake test".
    autofill: >
      Look in the build scripts for a non-empty test command.
      E.g., look for a Makefile (including Makefile.am) with a non-empty
      "check" or "test" entry.  Similarly look for an entry for
      maven (mvn test) or rake (rake test).
- test_most:
    category: SUGGESTED
    description: >
      It is SUGGESTED that the test suite cover most (or ideally all)
      the code branches, input fields, and functionality.
    autofill: >
      We could run a test suite with coverage enabled, but that would
      take far too long for our time budget.
      However, we *could* look for a Coveralls or Codecov badge
      (a link to a badge under <https://coveralls.io>) and pull in
      *that* data (since that would be the final result).
      Coveralls or Code prefer 90% or more statement coverage.
- test_continuous_integration:
    category: SUGGESTED
    description: >
      It is SUGGESTED that the project implement continuous integration
      (where new or changed code is frequently integrated into a central
      code repository and automated tests are run on the result).
    autofill: >
      We could look for evidence of pulls each of which are "relatively"
      small (instead of rare massive changes being the norm).
      We also see if it has a badge from CircleCI.
      Also, make this depend on the "test" or "test_invocation" criterion.
# New functionality testing
- test_policy:
    category: MUST
    description: >
      The project MUST have a general policy (formal or not) that as major
      new functionality is added, tests of that functionality SHOULD
      be added to an automated test suite.
    autofill: >
      Look for text patterns hinting at this in README, CONTRIBUTING,
      or the doc/* directory.
      Also, make this depend on the "test" or "test_invocation" criterion.
- tests_are_added:
    category: MUST
    description: >
      The project MUST have evidence that such tests are being added in
      the most recent major changes to the project.
    details: >
      Major functionality would typically be mentioned in the ChangeLog.
      (Perfection is not required, merely evidence
      that tests are typically being added in practice.)
    autofill: >
      Look for evidence that new tests are created.
      E.G., do at least some contributions include new tests
      (e.g., changes to contents in a directory whose full pathname
      contains the phrase "test").
- tests_documented_added:
    category: SUGGESTED
    description: >
      It is SUGGESTED that this policy on adding tests be
      <i>documented</i> in the instructions for change proposals.
    details: >
      However, even an informal rule is acceptable as long as the tests
      are being added in practice.
    autofill: >
      Look in CONTRIBUTING and README.  See also test_policy.
# Warning flags
- warnings:
    category: MUST
    na_allowed: true
    description: >
      The project MUST enable one or more compiler warning flags, a "safe"
      language mode, or use a separate "linter" tool to look for code quality
      errors or common simple mistakes, if there is at least one FLOSS tool that
      can implement this criterion in the selected language.
    details: >
      Examples of compiler warning flags include gcc/clang "-Wall".
      Examples of a "safe" language mode include Javascript "use strict"
      and perl5's "use warnings".
      A separate "linter" tool is simply a tool that examines the
      source code to look for code quality errors or common simple mistakes.
    autofill: >
      See the "details" - search the build script and source code for these.
- warnings_fixed:
    category: MUST
    na_allowed: true
    description: >
      The project MUST address warnings.
    details: >
      The project should fix warnings or mark them in the source code
      as false positives.  Ideally there would be no warnings, but a project
      MAY accept some warnings (typically less than 1 warning per 100
      lines or less than 10 warnings).
    autofill: >
      The only good way to do this is to actually do a build.
      If it uses CircleCI we could look at that.
- warnings_strict:
    category: SUGGESTED
    na_allowed: true
    description: >
      It is SUGGESTED that projects be maximally strict with warnings,
      but this is not always practical.
    autofill: >
      If we find "-Wall -Wextra" in the build that's pretty strict.
#
# SECURITY
#
# Secure development knowledge
- know_secure_design:
    category: MUST
    description: >
      The project MUST have at least one primary developer who knows
      how to design secure software.
    details: >
      This requires understanding the following
      design principles, including the 8 principles from <a
      href="http://web.mit.edu/Saltzer/www/publications/protection/">Saltzer
      and Schroeder</a>:
      <ul>
      <li>economy of mechanism (keep the design as simple and small as
      practical, e.g., by adopting sweeping simplifications)
      <li>fail-safe defaults (access decisions should deny by default, and
      projects' installation should be secure by default)
      <li>complete mediation (every access that might be limited must be
      checked for authority and be non-bypassable)
      <li>open design (security mechanisms should not depend on attacker
      ignorance of its design, but instead on more easily protected and changed
      information like keys and passwords)
      <li>separation of privilege (ideally, access to important objects
      should depend on more than one condition, so that defeating
      one protection system won't enable complete access.
      E.G., multi-factor authentication, such as requiring both a password
      and a hardware token, is stronger than single-factor authentication)
      <li>least privilege (processes should operate with the least privilege
      necessary)
      <li>least common mechanism (the design should minimize the mechanisms
      common to more than one user and depended on by all users, e.g.,
      directories for temporary files)
      <li>psychological acceptability (the human interface must be designed
      for ease of use - designing for "least astonishment" can help)
      <li>limited attack surface (the attack surface - the set of the different
      points where an attacker can try to enter or extract data - should
      be limited)
      <li>input validation with whitelists (inputs should typically be checked
      to determine if they are valid before they are accepted; this validation
      should use whitelists (which only accept known-good values),
      not blacklists (which attempt to list known-bad values)).
      </ul>
      A "primary developer" in a project is anyone who is familiar with
      the project's code base, is comfortable making changes to it, and is
      acknowledged as such by most other participants in the project.
      A primary developer would typically make a number of contributions
      over the past year (via code, documentation, or answering questions).
      Developers would typically be considered primary developers if they
      initiated the project (and have not left the project more than three
      years ago), have the option of receiving information on a private
      vulnerability reporting channel (if there is one), can accept commits
      on behalf of the project, or perform final releases of the project
      software.
      If there is only one developer, that individual is the primary developer.
    autofill: >
      We could try to search the documentation for evidence of phrases
      that suggest security knowledge, such as the phrases listed above.
- know_common_errors:
    category: MUST
    description: >
      At least one of the primary developers MUST know of common kinds of
      errors that lead to vulnerabilities in this kind of software,
      as well as at least one method to counter or mitigate each of them.
    details: >
      Examples (depending on the type of software)
      include SQL injection, OS injection, classic buffer overflow,
      cross-site scripting, missing authentication, and missing authorization.
      See the <a href="http://cwe.mitre.org/top25/">CWE/SANS top 25</a> or
      <a href="https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project">OWASP Top 10</a>
      for commonly used lists.
    autofill: >
      We could try to search the documentation for evidence of phrases
      that suggest security knowledge, such as the phrases for common
      types of vulnerabilities such as CWE/SANS top 25 or OWASP top 10.
      Simply mentioning those could also be an indicator.
# Use basic good cryptographic practices
- crypto_published:
    category: MUST
    na_allowed: true
    description: >
      The project's cryptographic software MUST use by default
      only cryptographic protocols
      and algorithms that are publicly published and reviewed by experts.
    # autofill: TODO
- crypto_call:
    category: SHOULD
    na_allowed: true
    description: >
      If the project software is an application or library, and its primary
      purpose is not to implement cryptography, then it SHOULD only call on
      software specifically designed to implement cryptographic functions;
      it SHOULD NOT re-implement its own.
    # autofill: TODO
- crypto_floss:
    category: MUST
    na_allowed: true
    description: >
      All functionality that depends on cryptography MUST be implementable
      using FLOSS.
    details: >
      See the <a href="https://opensource.org/osr">Open Standards Requirement
      for Software by the Open Source Initiative</a>.
    # autofill: TODO
- crypto_keylength:
    category: MUST
    na_allowed: true
    description: >
      The default keylengths MUST meet the NIST minimum requirements at
      least through the year 2030 (as stated in 2012).
    details: >
      These minimum bitlengths are: symmetric key 112, factoring modulus 2048,
      discrete logarithm key 224,
      discrete logarithmic group 2048, elliptic curve 224,
      and hash 224 (password hashing is not covered by this
      bitlength, more information on password hashing can be found in the
      <a href="#crypto_password_storage">crypto_password_storage</a> criterion).
      See <a href="http://www.keylength.com">http://www.keylength.com</a>
      for a comparison of keylength recommendations from various organizations.
      The software MUST be configurable so that it will
      reject smaller keylengths.
      The software MAY allow smaller keylengths in some configurations
      (ideally it would not, since this allows downgrade attacks,
      but shorter keylengths are sometimes necessary for interoperability.)
    # autofill: TODO
- crypto_working:
    category: MUST
    na_allowed: true
    description: >
      The default project security mechanisms MUST NOT depend on cryptographic
      algorithms that are broken (e.g., MD4, MD5, single DES, RC4,
      or Dual_EC_DRBG).
    # autofill: TODO
- crypto_weaknesses:
    category: SHOULD
    na_allowed: true
    description: >
      The project security mechanisms SHOULD NOT by default depend on
      cryptographic algorithms with known serious weaknesses (e.g., SHA-1).
    # autofill: TODO
- crypto_pfs:
    category: SHOULD
    na_allowed: true
    description: >
      The project SHOULD implement perfect forward secrecy for key agreement
      protocols so a session key derived from a set of long-term keys cannot
      be compromised if one of the long-term keys is compromised in the future.
    # autofill: TODO
- crypto_password_storage:
    category: MUST
    na_allowed: true
    description: >
      If passwords are stored for authentication of external users, the
      project MUST store them as iterated hashes with a per-user salt by
      using a key stretching (iterated) algorithm (e.g., PBKDF2, Bcrypt
      or Scrypt).
    # autofill: TODO
- crypto_random:
    category: MUST
    na_allowed: true
    description: >
      The project MUST generate all cryptographic keys and nonces
      using a cryptographically secure random number generator,
      and MUST NOT do so using generators that are not cryptographically secure.
    details: >
      A cryptographically secure random number generator may be a
      hardware random number generator, or it may be
      a cryptographically secure pseudo-random number generator (CSPRNG) using
      an algorithm such as Hash_DRBG, HMAC_DRBG, CTR_DRBG, Yarrow, or Fortuna.
    # autofill: TODO
# Secured delivery against man-in-the-middle (MITM) attacks
- delivery_mitm:
    category: MUST
    description: >
      The project MUST use a delivery mechanism that counters MITM attacks.
      Using https or ssh+scp is acceptable.
    details: >
      An even stronger mechanism is releasing the software with digitally
      signed packages, since that mitigates attacks on the distribution
      system, but this only works if the users can be confident that the
      public keys for signatures are correct <i>and</i> if the users will
      actually check the signature.
    autofill: >
      Look for a download site using https.
- delivery_unsigned:
    category: MUST
    description: >
      A cryptographic hash (e.g., a sha1sum) MUST NOT be retrieved over http and
      used without checking for a cryptographic signature.
    details: >
      These hashes can be modified in transit.
    autofill: >
      Again, look for a download site using https.
# Publicly known Vulnerabilities fixed
- vulnerabilities_fixed_60_days:
    category: MUST
    description: >
      There MUST be no unpatched vulnerabilities of medium or high severity that
      have been publicly known for more than 60 days.
    details: >
      The vulnerability must be patched and released by the project itself
      (patches may be developed elsewhere).  A vulnerability becomes
      publicly known (for this purpose) once it has a CVE with
      publicly released non-paywalled information (reported, for example,
      in the <a href="https://nvd.nist.gov/">National Vulnerability
      Database</a>) or when the project has been informed and the
      information has been released to the public (possibly by the project).
      A vulnerability is medium
      to high severity if its <a href="https://nvd.nist.gov/cvss.cfm">CVSS
      2.0</a> base score is 4 or higher.
      <br><br>
      <b>Note</b>: this means that users might be left vulnerable to all
      attackers worldwide for up to 60 days.  This criterion is often
      much easier to meet than what Google recommends in <a
      href="https://security.googleblog.com/2010/07/rebooting-responsible-disclosure-focus.html">Rebooting
      responsible disclosure</a>, because Google recommends that the 60-day
      period start when the project is notified <em>even</em> if the report
      is not public.<br>
      <b>Rationale</b>: We intentionally chose to start measurement from the
      time of public knowledge,
      and not from the time reported to the project, because this is much
      easier to measure and
      verify by those outside the project.
    autofill: >
      Look at vulnerability databases (such as NVD), pull out unpatched ones,
      and look at days since report.
- vulnerabilities_critical_fixed:
    category: SHOULD
    description: >
      Projects SHOULD fix all critical vulnerabilities rapidly after they
      are reported.
    autofill: >
       Look at vulnerability databases for say the last 2 years, and
       find the worst-case time for response to any critical vulnerabilities.
       More than 60 days is not a good sign.
# Other security issues
- no_leaked_credentials:
    category: MUST
    description: >
      The public repositories MUST NOT leak a valid private credential
      (e.g., a working password or private key) that is intended to limit
      public access.
    details: >
      A project MAY leak "sample" credentials for testing and
      unimportant databases, as long as they are not intended to limit
      public access.
    autofill: >
      Search repo for filenames that suggest credential leaking, e.g.,
      id_dsa (SSH private key).  Could also look at file contents for
      things like Amazon keys or Heroku keys.
      Could look at .env file, though "SECRETS" there might not really
      be secrets.
#
# ANALYSIS
#
# Static code analysis
- static_analysis:
    category: MUST
    na_allowed: true
    description: >
      At least one static code analysis tool MUST be applied to any proposed
      major production release of the software before its release,
      if there is at least one FLOSS tool that implements this criterion in
      the selected language.
    details: >
      A static code analysis tool examines the software code (as source
      code, intermediate code, or executable) without executing it
      with specific inputs.  For purposes of this criterion, compiler
      warnings and "safe" language modes do not count as static code
      analysis tools (these typically avoid deep analysis because
      speed is vital).  Examples of such static code analysis tools
      include <a href="http://cppcheck.sourceforge.net/">cppcheck</a>,
      <a href="http://clang-analyzer.llvm.org/">clang static analyzer</a>,
      <a href="http://findbugs.sourceforge.net/">FindBugs</a> (including <a
      href="https://h3xstream.github.io/find-sec-bugs/">FindSecurityBugs</a>),
      <a href="https://pmd.github.io/">PMD</a>,
      <a href="http://brakemanscanner.org/">Brakeman</a>,
      <a href="https://scan.coverity.com/">Coverity Quality Analyzer</a>, and
      <a href="http://www8.hp.com/au/en/software-solutions/static-code-analysis-sast/index.html">HP
      Fortify Static Code Analyzer</a>.
      Larger lists of tools can be found in places such as the
      <a href="https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis">Wikipedia
      list of tools for static code analysis</a>,
      <a href="https://www.owasp.org/index.php/Static_Code_Analysis">OWASP
      information on static code analysis</a>,
      <a href="http://samate.nist.gov/index.php/Source_Code_Security_Analyzers.html">NIST
      list of source code security analyzers</a>, and
      <a href="http://www.dwheeler.com/essays/static-analysis-tools.html">Wheeler's
      list of static analysis tools</a>.
      The <a href="https://continuousassurance.org/">SWAMP</a> is a no-cost
      platform for assessing vulnerabilities in software using a variety
      of tools.
    autofill: >
      Look in build scripts for execution of common tools, and in
      documentation for names of tools and the URL of a Coverity scan entry.
- static_analysis_common_vulnerabilities:
    category: SUGGESTED
    na_allowed: true
    description: >
      It is SUGGESTED that at least one of the static analysis tools
      used for the static_analysis criterion
      include rules or approaches to look for
      common vulnerabilities in the analyzed language or environment.
    autofill: >
      We might start by looking primarily for tools that also meet this.
      E.g., brakeman for Ruby on Rails.
- static_analysis_fixed:
    category: MUST
    na_allowed: true
    description: >
      All medium and high severity exploitable vulnerabilities
      discovered with static code analysis MUST be fixed
      in a timely way after they are confirmed.
    details: >
      A vulnerability is medium to high severity if its <a
      href="https://nvd.nist.gov/cvss.cfm">CVSS 2.0</a> is 4 or higher.
    # autofill: TODO
- static_analysis_often:
    category: SUGGESTED
    na_allowed: true
    description: >
      It is SUGGESTED that static source code analysis occur on every
      commit or at least daily.
    autofill: >
      Look for commit hooks or continuous integration tools like CircleCI
      that would meet this.
# Dynamic code analysis
- dynamic_analysis:
    category: SUGGESTED
    description: >
      It is SUGGESTED that at least one dynamic analysis tool be applied
      to any proposed major production release of the software before
      its release.
    details: >
      A dynamic analysis tool examines the software
      by executing it with specific inputs.
      For example, the project MAY use a fuzzing tool (e.g.,
      <a href="http://lcamtuf.coredump.cx/afl/">American
      Fuzzy Lop</a>) or a web application scanner (e.g.,
      <a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project">OWASP
      ZAP</a> or <a href="http://w3af.org/">w3af</a>).
      For purposes of this criterion the dynamic analysis
      tool needs to vary the inputs in some way to look for
      various kinds of problems <em>or</em> be an automated
      test suite with at least 80% branch coverage.  The
      <a href="https://en.wikipedia.org/wiki/Dynamic_program_analysis">Wikipedia
      page on dynamic analysis</a> and the
      <a href="https://www.owasp.org/index.php/Fuzzing">OWASP page on
      fuzzing</a> identify some dynamic analysis tools.
      <br>
      The analysis tool(s) MAY be focused on looking for security
      vulnerabilities, but this is not required.
      <b>Rationale:</b> Static source code analysis and dynamic analysis tend
      to find different kinds of defects (including defects that lead to
      vulnerabilities), so combining them is more likely to be effective.
    autofill: >
      Look in documentation for references to such tools.
- dynamic_analysis_unsafe:
    category: SUGGESTED
    na_allowed: true
    description: >
      It is SUGGESTED that if the software is application-level software
      written using a memory-unsafe language (e.g., C or C++) then at
      least one dynamic tool (e.g., a fuzzer or web application scanner)
      be routinely used with a mechanism to detect memory safety problems
      such as buffer overwrites.
    details: >
      Examples of mechanisms to detect memory safety problems
      include Address Sanitizer (ASAN) and
      <a href="http://valgrind.org/">valgrind</a>.
      Widespread assertions would also work.
      If the software is not application-level,
      or is not in a memory-unsafe language, then this criterion is
      automatically met.
    autofill: >
      Look in build/test script for reference to invocation of
      valgrind or ASAN.
- dynamic_analysis_enable_assertions:
    category: SUGGESTED
    description: >
      It is SUGGESTED that the software include many run-time assertions
      that are checked during dynamic analysis.
    autofill: >
      Perhaps look in source code for many asserts.
- dynamic_analysis_fixed:
    category: MUST
    description: >
      All medium and high severity exploitable vulnerabilities
      discovered with dynamic code analysis MUST be fixed
      in a timely way after they are confirmed.
    details: >
       A vulnerability is medium to high severity if its <a
       href="https://nvd.nist.gov/cvss.cfm">CVSS 2.0</a> base score is 4
    # autofill: TODO
#
# FUTURE
#
- installation_common: # Put after 'english'
    category: SHOULD
    future: true
    description: >
      The project SHOULD
      provide a way to easily install and uninstall the software using a
      commonly-used convention.
    details: >
      Examples include using a package manager (at
      the system or language level), "make install/uninstall" (supporting
      DESTDIR), a container in a standard format,
      or a virtual machine image in a standard format.
      The installation and uninstallation process (e.g., its packaging)
      MAY be implemented by a third party as long as it is FLOSS.
    autofill: >
      Look for a standard install format, or a build instruction for one.
- build_reproducible: # After build_floss_tools
    category: SUGGESTED
    future: true
    na_allowed: true
    description: >
      It is SUGGESTED that
      the project have a
      <a href="https://reproducible-builds.org/">reproducible build</a>.
    details: >
      With reproducible builds, multiple parties can independently redo the
      process of generating information from source files and get exactly
      the same result.
      The <a href="https://reproducible-builds.org/docs/">reproducible builds project has documentation on how to do this</a>.
      This criterion does not apply if no building occurs
      (e.g., scripting languages where the source code
      is used directly instead of being compiled).
    autofill: >
      If the project seems to only have scripting language code,
      prefill it as N/A.
      Otherwise, check if the reproducible build project lists the project
      as being reproducible.
      The only *real* way to check is to actually perform the build
      multiple times, but we don't have time for that.
- crypto_used_network:
    category: SHOULD
    future: true
    na_allowed: true
    description: >
      The project SHOULD NOT use
      unencrypted network communication protocols (such as HTTP and telnet)
      if there an encrypted equivalent (e.g., HTTPS/TLS and SSH),
      unless the user specifically requests or configures it.
- crypto_tls12:
    category: SHOULD
    future: true
    na_allowed: true
    description: >
      The project SHOULD,
      if it supports TLS, support at least TLS version 1.2.
      Note that the predecessor of TLS was called SSL.
- crypto_certificate_verification:
    category: MUST
    future: true
    na_allowed: true
    description: >
      The project MUST,
      if it supports TLS, perform TLS certificate verification by default
      when using TLS, including on subresources.
    details: >
      Note that incorrect TLS certificate verification is a common mistake.
      For more information, see
      <a href="http://crypto.stanford.edu/~dabo/pubs/abstracts/ssl-client-bugs.html">"The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software" by Martin Georgiev et al.</a>
      and
      <a href="https://blogs.gnome.org/mcatanzaro/2016/03/12/do-you-trust-this-application/">"Do you trust this application?" by Michael Catanzaro</a>.
- crypto_verification_private:
    category: SHOULD
    future: true
    na_allowed: true
    description: >
      The project SHOULD,
      if it supports TLS, perform certificate verification
      before sending HTTP headers with private information
      (such as secure cookies).
- hardened_site: # After delivery_mitm?
    category: SUGGESTED
    future: true
    description: >
      It is SUGGESTED that the project website, repository (if accessible
      via the web), and download site (if separate) include key hardening
      headers with nonpermissive values.
    details: >
      Note that GitHub is known to meet this.
      Sites such as https://securityheaders.io/ can quickly check this.
      The key hardening headers are:
      Content Security Policy (CSP), HTTP Strict Transport Security
      (HSTS), X-Content-Type-Options (as "nosniff"), X-Frame-Options,
      and X-XSS-Protection.
    autofill: >
      Load the URLs and look for those HTTP headers.
- hardening: # In 'Other security issues'
    category: SUGGESTED
    future: true
    description: >
      It is SUGGESTED that hardening mechanisms be used so software defects
      are less likely to result in security vulnerabilities.
    details: >
      Hardening mechanisms may include
      HTTP headers like Content Security Policy (CSP),
      compiler flags to mitigate attacks
      (such as -fstack-protector), or compiler flags to
      eliminate undefined behavior.
      For our purposes least privilege is not considered a hardening
      mechanism (least privilege is important, but separate).
    autofill: >
      Look for gems like secure_headers, and
      relevant compiler flags in build files.