Open source creates value, but how do you measure it?

Open source software has transformed our lives. Using someone else’s running code allows developers to focus efforts on unsolved problems without reinventing the wheel—accelerating innovation at scale. To developers, this is obvious. But too often, it is not as clear to policymakers. This matters, because while governments often provide public goods, digital infrastructure can be overlooked. And it’s not just missed opportunities to do more: policies may inadvertently endanger open source collaboration. One of our primary goals on the GitHub Policy Team is to help policymakers understand the value of open.

There’s been some recent research estimating that open source drove between €65 and 95 billion of European GDP in 2018 alone.^[1] Within months, this research already had an impact on policy. The European Commission cited the study in establishing new rules to streamline the process to open source its software.

We need more, and there are many open questions. Here, I outline three themes that we at GitHub Policy think should drive this conversation with policymakers—to help open source prosperity and sustainability. These themes are only a starting point: if you have thoughts, please get in touch.

What is open source’s macroeconomic impact?

Policymakers need society-wide analysis to inform society-wide decisions. Research on the open source ecosystem at national and larger scales finds increases in GDP,^{[1, 2]} labor productivity,^[1] and start-up formation.^[3] These types of studies help demonstrate to policymakers and others that contributions to open source bring real benefits to local economies.

Yet there are important questions that are still unanswered:

• Leading studies have focused on the EU; what do these impacts look like for the US, India, and the African continent, among other regions around the world? Are there distributional effects, for example, where some countries may benefit more than others?
• Many studies demonstrate promising correlations between open source and economic growth. Yet what if causality is reversed, and open source is a byproduct of growth? More research that exploits natural experiments,^[4] where developers were cut-off from open source communities or policies that then shifted open source activity, would help bolster the case that open source drives economic benefits.
• With the benefit of hindsight, were past studies’ predictions correct, overly optimistic, or perhaps they underestimated open source’s impact? Research offered projections that can be evaluated with the benefit of hindsight: a 10% increase in EU contributors would generate 0.4–0.6% greater GDP for the single market^[1]; estimated costs of creating and maintaining Debian 3.1 through 2010 would be €100 billion^[2]; a 1% increase in commits in a country is associated with 0.6–1.2% additional new startups.^[3]

What benefit comes from an individual open source project?

Large-scale impacts are ultimately driven by the aggregate effects of countless open source projects. We need a better picture of the value at this smaller scale too. Existing research tends to value open source software by estimating the costs associated with its creation,^[5] cost savings from substituting for paid software,^{[6, 7]} or using case studies to describe innovations enabled by certain projects.^[8] More directly linking to value, some studies look at revenues of firms that contribute to open source.^[2] But the picture remains incomplete.

There are several promising research directions to measure the value of modern open source development frameworks and to quantify our favorite xkcd:

• How should we measure the impact of individual software projects within dependencies? A dependency graph says much about the value of a particular software project embedded within it. This type of research is already being pursued in security and some preliminary economics work^{[5, 7]} but should be expanded.
• How do we account for the counterfactual impact of packages? Value may not be a function of the dependency graph alone. If a developer could easily write a function to replace a highly used package, for example is-odd, network analysis may overstate its importance.
• How can we link measures of economic value, including firm investment and maintainers’ income from open source funding efforts, to dependencies? Might hedonic pricing methods used elsewhere in economics help identify the particular value of individual projects and their qualities within the dependency graph?
• How should maintainers get paid? These research questions can inform needed improvements in open source sustainability by creating more precise measures of value that can be tied to maintainers’ work and inform compensation models. They can also support models to guide government investment in securing open source.

How should open innovation be measured?

Deepening our understanding of the value of contemporary open source development raises another related question: how do we understand its relationship with innovation? Surveys dating back more than a decade have found that companies consider open source to be a way to generate new ideas, more so than reviewing patents.^[2] Yet, patents continue to be widely used as a metric of innovation, while open source is neglected. Part of this is a function of difficulties in measurement. GitHub data may help. Innovation output measures could look at forks and stars on projects. We are working to improve our metrics to better support research here, and will have more to share soon.

Beyond updating our understanding of innovation outputs with open source, there are many more innovation questions:

• How does open source software contribute to innovation as an input, and can targeted research funding for open source increase this contribution? Further research should build on initial measurement efforts^[7] to understand how and to what extent open source software accelerates scientific research.
• As open source business models have evolved over time, how have firm contributions to open source changed? Amid these business innovations, particularly the rise of cloud-based software as a service, what is the relative contribution to open source from these big cloud companies?
• How do we value the contributions of innovations in developer tools to open source, including maintainers’ productivity and workload? These tools include GitHub Actions and GitHub Copilot, but also extend well beyond our platform.
• What is the economic impact—at both an organizational and economy-wide level—of new institutional approaches to open source, including the Open Source Program Office, pioneered in industry that is now percolating into the public and social sectors?

How can you help?

What is not measured, all too often, is invisible. The GitHub Policy Team intends to improve our collective understanding of open source economics to better optimize policy. Economics is far from the only lens to view the impact of open source. For example, open source has important affordances for transparency, trust, and inclusive design. That said, economics is particularly important to policymakers, and improving policy is our primary goal.

These questions are invitations. Please get in touch with comments, sources, and further questions that you’d like to see answered. Researchers, get in touch if you’re interested in making proposals. Ultimately, we hope to support research to help policymakers understand the value of open. Watch this space.

Follow GitHub Policy on Twitter for updates about the laws and regulations that impact developers.

[1] Blind, et al. (2021). The impact of open source software and hardware on technological independence, competitiveness and innovation in the EU economy. European Commission.

[2] Ghosh, et al. (2006). Study on the economic impact of open source software on innovation and the competitiveness of the Information and Communication Technologies (ICT) sector in the EU. UNU-MERIT.

[3] Wright, Nagle & Greenstein. (2021). Open source software and global entrepreneurship: A virtuous cycle. Harvard Business School Working Paper, 20-139.

[4] Nagle. (2019). Government technology policy, social value, and national competitiveness. Harvard Business School Working Paper, 19-103.

[5] Robbins, et al. (2018). Open source software as intangible capital: Measuring the cost and impact of free digital tools. 6th International Monetary Fund Statistical Forum.

[6] Greenstein & Nagle. (2014). Digital dark matter and the economic contribution of Apache. Research Policy, 43(4), 623-631.

[7] Keller, et al. (2018). Opportunities to observe and measure intangible inputs to innovation: Definitions, operationalization, and examples. Proceedings of the National Academy of Sciences, 115(50), 12638-12645.

[8] Christensen, Ghose & Mathur. (2020). Economic impact of open source software on competition, innovation, and development in India. National Conference on Economics of Competition Law 2020, Competition Commission of India.

Written by

Peter Cihon

@pcihon