Making open source data more available
Data gives us insight into how people build software, and the activities of open source communities on GitHub represent one of the richest datasets ever created of people working together…
Data gives us insight into how people build software, and the activities of open source communities on GitHub represent one of the richest datasets ever created of people working together at scale.
In 2012, the community led project, GitHub Archive was launched, providing a glimpse into the ways people build software on GitHub. Today, we’re delighted to announce that, in collaboration with Google, we are releasing a collection of additional BigQuery tables to expand on the data from that project1.
This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains activity data for more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions.
With this new dataset, it’s a simple query to find out which are the most commonly used Go packages, which US-schools have the most open source contributors and find all of the things that should never happen.
Just as books capture thoughts and ideas, software encodes human knowledge in a machine-readable form. This dataset is a great start toward the pursuit of documenting the open source community’s vast repository of knowledge—but there’s more to be done. Over the coming months, you can expect to hear from us on how we hope to make open source data even more available, portable, and useful.
Whether you’re a researcher studying open source communities, an organization looking to monitor the health of your open source projects, or curious about the latest trends in software development, go check out the new dataset hosted on Google Cloud to analyze one of the largest datasets of people collaborating on the planet.
1. If you’d like to hear more about the data release then check out this episode of The Changelog.
Tags:
Written by
Related posts
![](https://github.blog/wp-content/uploads/2023/10/Collaboration-DarkMode-2.png?resize=400%2C212)
2024 is the biggest global election year in history. What’s at stake for developers?
GitHub is considering what is at stake for our users and platform, how we can take responsible action to support free and fair elections, and how developers contribute to resilient democratic processes.
![](https://github.blog/wp-content/uploads/2023/10/AI-DarkMode-2.png?resize=400%2C212)
GitHub named a Leader in the Gartner first-ever Magic Quadrant for AI Code Assistants
This year, as part of its annual Magic Quadrant series, Gartner published a first-of-its-kind report analyzing the state of play in the AI Code Assistants market–and named GitHub a Leader.
![AI in software development 2024 survey results with colorful graphics and percentages.](https://github.blog/wp-content/uploads/2024/08/ai-survey-header.png?resize=400%2C212)
Survey: The AI wave continues to grow on software development teams
We surveyed 2,000 people on software development teams at enterprises in the U.S., Brazil, India, and Germany about the use, experience, and expectations around generative AI tools in software development.