Welcome to the CHAOSS Metrics repository. CHAOSS Metrics repository captures metrics for assessing open source community health and sustainability. Such metrics are aimed at understanding project diversity & inclusion, growth-maturity-decline, risk, and value. For more information about the CHAOSS project go to our website at: https://chaoss.community/
(1) Capturing metrics based on the work of community members who have participated at CHAOSS events, worked in the repo, and discussed through on email. We capture the metrics that people find meaningful to their particular contexts when understanding project health and sustainability. In this, we work to represent metrics through concise definitions, known uses cases, sample visualizations, and sample implementations. The CHAOSS Metrics repository captures all potential metrics. It is the WGs that make these metrics meaningful, leading to our second goal.
(2) Participating with the CHAOSS workgroups. Through these workgroups, we consolidate metrics in meaningful ways and inform the metrics by questioning how to capture and deploy (or not) them:
CHAOSS Common Metrics Working Group
CHAOSS Diversity & Inclusion Working Group
CHAOSS Evolution Working Group
- Risk Mitigation
- Track Corporate Engagement
- Identify Sustainable Projects
- Identify Single Points of Failure
- Avoid In-take of an Inactive Project
- Identify Open Source Projects that Need Support
- Assess Value Generated through Community and Engagement
- Show that Active Community Management Bears Desired Results
- Project Maturity
- Project Viability
- Growth of Community
- Momentum of Community
- Diversity of Community
- Timeliness of Maintainers
- Attentiveness of Maintainers
- Activity Level - Responsiveness
- Distribution of Code Contributions
- Vanity metrics (might have use in other cases, e.g. stars)
- Ecosystem Health (upstream, downstream, and related projects)
- Aggregate Project-tree Health (combined health metrics of all linked dependencies)
- Value Derivation
- Style of Project
- Project Comparison
- Maturity of Project
- Programming Language
- Quality of Ecosystem
- Community Composition
The following is a full list of identified metrics. How the metrics live in practice is work that happens in the workgroups.
Please note that there is a folder called "metrics_scratchpad" underneath the activity-metrics folder. This directory is for metrics that are nascent ideas but have not been through our metric naming and classification process yet. We are trying to name all the metrics according to the types of data they characterize. For example, metrics related to "code" are prefixed with "code-", etc.
| Name | Question/Description | Deployment in CHAOSS Workgroup |
|---|---|---|
| Accepted Code Contributions | Percentage of new contributor code versus total code over time. | |
| Age of Community | Time since repository/organization was registered; or time since first release (Chengalur-Smith et al., 2010; Grewal, Lilien, & Mallapragada, 2006). | |
| All Licenses | List of licenses. | |
| Alternatives | Are there a variety of communication channels? | D&I |
| Apache Maturity Model | The Apache Project Maturity Model guidelines for assessing the maturity of a project. | |
| Attendees Demographics | How diverse are the attendees? | D&I |
| Availability of Add-on Products | Availability of 3rd party plug-ins, modules, utilities, etc. that provide additional functionality for the project's software. | |
| Average Issue Resolution Time | The average amount of time it takes for issues to be closed. | |
| Average Time of First Maintainer Response to Code Merge Request | The average amount of time it takes for a maintainer to make the first response to a code merge request. | |
| Average Time of First Response to Issue | The average amount of time it takes for the first response to an issue. | |
| Average Time of Open Issues | The average amount of time open issues have remained open. | |
| Board/Council Diversity | What is the diversity within our governing board or council? | D&I |
| Blog posts | Number of blog posts that mention the project. | |
| Bug Age | Age of known bugs in issue tracker. | |
| Bugs after Release | Number of bugs reported after a release. | |
| Bus Factor | The number of developers/organizations it would need to lose to destroy its progress. | Risk |
| Captioning | Do we provide text captioning for spoken communication? | D&I |
| Change in Maintainer Number | Number of maintainers added/removed over time. | |
| CII Best Practices | The CII Best Practices Badge Program provides maturity self-certification: passing, silver, and gold. | Risk |
| Closed Issues | What is the number of closed issues? | Evolution |
| Closed Issues New Contributors | What is the number of persons closing an issue for the first time? | Evolution |
| Closed Issue Resolution Duration | What is the duration of time for issues to be resolved? | Evolution |
| Code Commits | What is the number of code commits? | |
| Code of Conduct at Event | How does the Code of Conduct for events support diversity and inclusion? | D&I |
| Code of Conduct Enforcement | Is enforcement process running at scale (volume, responsiveness, accuracy, fairness)? | D&I |
| Code Merge Duration | What is the duration of time between code merge request and code commit? | |
| Code Modularity | Modular code allowing for parallel development (Baldwin & Clark, 2006). | |
| Code Review Efficiency | What is the number of merged code changes/number of abandoned code change requests? | Evolution |
| Code Review Iteration | What is the number of iterations that occur before a merge request is accepted or declined? | Evolution |
| Collaboration Style | How inclusive is community collaboration? | D&I |
| Commercial Offerings | Availability of commercial products or services based on the project. | |
| Commit Bias | Acceptance rate and time to acceptance differences per gender, ethnicity, and relevant diversity characteristics. | |
| Commit Hours | Does a contributor commit during or outside of regular business hours? Triangulate UTC commit times from a contributor with locale of contributor and employer to determine regular business hours. | |
| Committers | The number of individuals who have committed code to a project. | Risk |
| Communication Channels | How welcoming, responsive, respectful are interactions even on hot topics of debate? What is the diversity of voices speaking/being heard? | D&I |
| Community Activity | Contribution Frequency. Contribution = commit, issue, comment, etc). | |
| Contributing Organizations | What is the number of contributing organizations? | Evolution |
| Contribution Age | Time since last contribution. Contribution = commit, issue, comment, etc.). | |
| Contribution Diversity | Ratio of code committed by contributors other than original project initiator. Contributions going up beyond the core team. | |
| Contribution Sentiment | What are the stars, thumbs up, sentiment in comments? | D&I |
| Contribution-Type | Coding, Quality Assurance/Testing, Localization/L10N, Diversity & Inclusion, Event Organization, Documenting, Community Building/Management, Teaching, Trouble-shooting/Support, Creative/Design, Social Media, Writing Articles, Bug Triaging, UI/UX, Security/Campaign Advocacy | D&I |
| Contribution Type | Does recognition skew to a particular kind of contribution? | D&I |
| Contribution Volume | Do we have a bias towards small contributions or multiple contributions? | D&I |
| Contributor Activity | Activity level of individual contributors. | |
| Contributor Breadth | Ratio of non-core committers (drive-by and peripheral committers). | |
| Contributor Country | Primary residence of the contributor. | |
| Contributor Demographics | Gender, age, location, education, and skills. | |
| Contributor Diversity | Ratio of contributors from a single company over all contributors. Also described as: Maintainers from different companies. Diversity of contributor affiliation. | |
| Contributor Employer Country | Headquarters location of the organization employing the contributor if they are contributing on behalf of their employer. | |
| Contributor Importance | Percentage of commits by individual contributors from identified organizations. | |
| Contributor Seniority | For each active contributor, time since first contribution. Experienced contributors providing value to the community, since they carry with them (in part) the history of the project. | |
| Contributors | What is the number of contributors? | Evolution |
| Copyright Declaration | The degree to which the project properly declares copyright ownership, including the copyright symbol or 'copyright' word, the year of the creation, the name of the author, and a rights statement. | |
| Country | Country location of contributor. | |
| Decision Distribution | Central vs. distributed decision making. Governance model, scalability of community. | |
| Dependency Depth | Number of projects included in code base + number of projects relying on focal project. Indicator about centrality in open source dependency network. | |
| Distribution of Work | How much is recent activity distributed? | |
| Diversity Access Tickets | Are Diversity Access Tickets offered for an event? | D&I |
| Documentation | What is the thoroughness, and accessibility of documentation according to a set of criteria? | D&I |
| Downloads of Non-software Artifacts | Number of downloads of non-software artifacts (e.g. documentation, sample apps, test suites, etc). | |
| Elephant Factor | The minimum number of companies whose employees perform 50% of the commits | Risk |
| Family Friendliness | Does the event empower those who care for children to attend? | D&I |
| File License Declarations | A list of license declarations on the software package files. | |
| First Response to Issue Duration | Time between a new issue is opened and a maintainer responds Also called: bug response rate. The maintainer is believed to not “pile on” but try to solve an issue | Evolution |
| Followers | Number of followers. | |
| Forks | Number of forks. | Evolution |
| Foundation Staff Diversity | What is the diversity of foundation staff? | D&I |
| Gatherings | Number of face-to-face/in-person meetings per year. | |
| Installs | Number of software installations of the project. | |
| Issue Comments | Number of comments per issue. | |
| Issue Resolution Efficiency | What is the number of closed issues/number of abandoned issues? | Evolution |
| Issues Submitted/Closed | Issues submitted vs. issues closed. Example. | |
| Issue Tracker | How well is a project issue tracker setup to invite new contributors, skilled contributors, non-technical contributors? | D&I |
| Job Postings | Number of job postings that mention the project as a preferred or required skill. | |
| Known Vulnerabilities | Number of reported vulnerabilities. Could be limited to issue-tracker or extended vulnerability databases (e.g. CVE). | |
| Language Bias | Bias against gender and ethnicity in use of language. | |
| Language Makeup | Makeup of a project in terms of whitespace, code, and comments. | |
| Leadership Demographics | Demographics of project's leadership (e.g. Board, Technical Steering Committee, Maintainers, etc.) over time. | |
| License Conflicts | Project containing incompatible licenses. | |
| License Count | Number of licenses. | Risk |
| License Coverage | Number of files with a file notice (copyright notice + license notice). | Risk |
| License Declared | What license does the project declare? | Risk |
| License Identification Methods | A list of methods or tools used for identifying licenses in files. | |
| Lines of Code Changed | What is the number of lines of code changed? | |
| Listening | How well do our mechanisms for listening accommodate community? | D&I |
| Maintainer Promotion | Last time a maintainer was added. | |
| Maintainer Response to Merge Request Duration | What is the duration of time for a maintainer to make a first response to a code merge request? | Evolution |
| Mode Alternatives | What alternative communication modes do we offer? | D&I |
| New Contributing Organizations | What is the number of new contributing organizations? | Evolution |
| New Contributions | Percentage of contributions (patches, pull requests, etc.) from new contributors vs all accepted contributions over time. | |
| New Contributors | What is the number of new contributors? | Evolution |
| New Contributors Closing Issues | What is the number of persons closing an issue for the first time? | Evolution |
| New Contributors of Code Reviews | What is the number of persons contributing with reviews of code for the first time? | Evolution |
| New Contributors of Commits | What is the number of persons contributing with an accepted commit for the first time? | Evolution |
| New Contributors of Initiated Code Reviews | What is the number of persons initiating a code review for the first time? | Evolution |
| New Contributors of Code Reviews | What is the number of persons contributing with reviews of code for the first time? | Evolution |
| New Contributors on the Email List | What is the number of persons posting messages in mailing lists for the first time? | Evolution |
| New Contributors* vs Maintainers** | Ratio of new contributors to maintainers over time. | |
| Non-source Contributions | Track contributions like running tests in test environment, writing blog posts, producing videos, giving talks, etc. | |
| Number of Active Users | Number of active users of the project. | |
| Number of Contributing Organizations | Number of organizations participating in the project over time. | |
| Onion Layers | Distance between onion model layers (users, contributors, committers, and steering committee). Rule of thumb: factor of 10x between layers. (OSLS'17 Node.js keynote). | |
| Open Issues New Contributors | What is the number of persons opening an issue for the first time? | Evolution |
| Open Issue Age | What is the the age of open issues? | Evolution |
| Package License Declaration | A list of license declarations on the software package. | |
| Paid Developers | Number of paid developers in community over time. | |
| Path to Influence | What opportunities are there to move into governance? | D&I |
| Path to Leadership | A communicated path from lurker to contributor to maintainer. | |
| Path to Maintainership | Path to maintainership published. | |
| People Opening Issues | How many people are opening issues? | |
| Project Life Cycle | Community assigned label. For example: proposal, incubaton, active, deprecated, end of life (Source: Hyperledger). | |
| Perceived Value | How are we valuing contributions and contribution types? | D&I |
| Percentile Distribution of First Maintainer Response to Code Merge Request | The proportional frequency of time it takes for a maintainer to make the first response to a code merge request. | |
| Percentile Distribution of First Response Time | The proportional frequency of time it takes for the first response to an issue. | |
| Percentile Distribution of Issue Resolution Time | Proportional frequency of closed issue time duration. | |
| Percentile Distribution of Open Issue Time | Proportional frequency of open issue time duration. | |
| Percentile Distribution of Time to Merge Code | Proportional frequency of code merge to upstream time duration. | |
| Pony Factor | The minimum number of developers performing 50% of the commits. The Math | |
| Pull Request Comments | Number of comments per pull request. | Evolution |
| Pull Request Comment Duration | The difference between the timestamp of the pull request creation date and the most recent comment on the pull request. | Evolution |
| Pull Request Discussion Diversity | Number of different people discussing each pull request. | |
| Pull Request Made vs. Closed | Pull requests made vs. pull requests closed Example. Encompasses number of pull requests rejected. | |
| Pull Requests Merged | Number of Code Commits | |
| Pull Requests Merged/Closed (Contribution Acceptance) | Ratio of contributions accepted vs. closed without acceptance. | Evolution |
| Pull Requests Open | Number of open pull requests. | Evolution |
| Pull Requests Over Time | How many pull requests have been submitted over a specified time period? | |
| Qualified Committers | Contributions over time and what components they commit to over time. | |
| Quick Links | No Question/Definition | D&I |
| Recognition Type | No Question/Definition | D&I |
| Recognition Value | Do different demographics value different types of recognition? | D&I |
| Relative Activity | Ratio of GH issues+comments, GH pull requests+comments, and GH commits for the project members and for the non-project members. Compare the activity between committers-as-a-group and contributors-as-a-group. It shows when a project is not yet popular or when a project is not paying attention to its users. | |
| Release Maturity | Ratio of major and minor releases. | |
| Release Note Completeness | Number of functionality changes and bug fixes represented in release notes vs. release. Good for users, also shows diligence of community. | |
| Release Velocity | Time between releases. Regular releases are a reliability indicator. | |
| Reopened Issues | Rate of issues closed but discussion continues or issues that were closed and re-opened. | |
| Repository Size | Overall size of the repository. | |
| Response Times & Quality | How quickly and well do we respond to suggestions, PRs, questions? | D&I |
| Retrospectives | Existence of after release meetings. Collect lessons learned, improve processes, recognize contributors. | |
| Reviews | What is new review requests for changes to the source code, during a certain period.? | Evolution |
| Review Efficiency | Number of merged patches / number of abandoned patches over time. | |
| Rewards | Rewards, shout-outs, recognition, and mentions in pull-requests or change logs. | |
| Roadmap | Existence and quality of roadmap. Best practice as community engagement and scalability (might not be automatically computable). | |
| Role Definitions | Existence and quality of role definitions. Relates to "Path to Leadership". | |
| Sentiment | What is the sentiment within external communication channels regarding our own press releases and within our internal communication channels, e.g., mail lists or IRC? | D&I |
| Size of Code Base | Lines of code. | |
| Software Downloads | Number of project software downloads. Beware: downloads might be skewed by builders. Used as measure for success (Grewal, Lilien, & Mallapragada, 2006). | |
| Software Bill of Materials | Provides a single source document that provides this information both for internal use and downstream distribution of software packages. | Risk |
| Speaker Demographics | How well does the speaker lineup for the event represent a diverse set of demographics? | D&I |
| Speaking | Are we speaking to our communities in an accessible range? | D&I |
| Stack Overflow | Number of questions asked, response rate, number of responding people that have verified solutions. | |
| Stars | Number of stars. | |
| Sub-Projects | What is the number of sub-projects? | Evolution |
| Team/Module Ownership Diversity | What is the diversity of other bureaucratic and administrative foundation teams, e.g. working groups, committees, or ambassador groups? | D&I |
| Test Coverage | Percentage of codebase covered by developer tests. | Risk |
| Time to Contributor | Time to becoming a contributor. | |
| Timezone | Timezone of contributor. | |
| Technical Jargon | Does the language skew to technical-confidence vs. technical ability? | D&I |
| Total Contributing Organizations | The total number of organizations contributing over time. | |
| Total Contributors | The total number of contributors over time. | |
| Total New Contributing Organizations | The total number of new organizations contributing over time. | |
| Total New Contributors | The total number of new contributors over time. | |
| Total Sub-projects | The total number of sub-projects over time. | |
| Transparency | Number of comments per issue. Discussion is occuring openly. | |
| Unity | Rivalry or unity of community. | |
| Update Age | Time since last update. | |
| Update Rate | Number of updates over time. | |
| Update Regularity | How consistently and frequently are updates provided. | |
| Use of Acronym | Frequency of acronyms used as a barrier for new contributors. | |
| User Dependency | Number of users who are aware that they depend on the software over time. | |
| User Groups | Number of user groups that perform a variety of crucial marketing, service support, and business-development functions at the grassroots level\ (Bagozzi & Dholakia, 2006) | |
| V-index | An index of project's first-order and second-order downstream dependencies. Example | |
| Watchers | Number of watchers. | |
| YouTube Videos | Number of Youtube videos that mention or specifically deal with the project (e.g. tutorials). |
Contribute and Participate. If you would like to propose a new metric to this repository, open a new issue and provide a pull request to open the discussion about inclusion. If you would like to provide metric details, we have a metric-template to be used. Follow the same approach regarding an issue/pull request to have your detail changes discussed and potentially merged.
All contributions to implementation-agnostic metrics and standards, including associated scripts, SQL statements, and documentation, will be received and made available under the MIT License (https://opensource.org/licenses/MIT).