Meet Nodeload, the new Download server
I recently launched the first GitHub node.js project: nodeload. Nodeload replaces the ruby git archive download server. Basically, any time you download a tarball or zip file of any repository…
I recently launched the first GitHub node.js project: nodeload. Nodeload replaces the ruby git archive download server. Basically, any time you download a tarball or zip file of any repository branch or tag, you’re going through nodeload.
![](https://github.blog/wp-content/uploads/2010/07/5bbd5272-b1fd-11e4-9c91-136fb38c48ad.png?resize=330%2C238)
This is a challenging issue because we can’t just store downloads anywhere. Either we pre-build archives for every single branch and tag after every repo commit, or we generate each one fresh. Of course, there is a healthy hot cache of built archives to deal with popular requests.
In the old system, requests to download an archive would create a Resque job. This Resque job actually runs a git archive
command on the archive server, pulling data from one of the file servers. Then, the original request sends you to a small ruby Sinatra app to wait for the job. It would essentially just check for the existence of a memcache flag before redirecting to the final download address. At the time, we were running about 3 Sinatra instances and 3 Resque workers.
This is a pragmatic solution when time and manpower are your constraints. I saw it as a great opportunity for Node.js. This is a simple feature that doesn’t depend on much of the site. In fact, the extra moving parts make it seem more complicated than it really is. The very nature of serving fresh git archives highlights the strengths of an event driven environment like Node.js, and the weaknesses of a blocking environment like Ruby.
However, I ran into a big roadblock right away: How do I even get the right file server for an archive? How can I authenticate whether a user has access to a private repo? Rather than reimplementing internal application logic, I went with a simple JSON API call. It returns the proper HTTP codes for missing or unauthorized repos, or a JSON object with the filesystem information. From there, nodeload spawns a git archive
process, and redirects the user when it’s finished. This way I was able to use the existing logic for user permissions and repository paths without duplicating logic or loading the GitHub app in ruby.
This is just the first iteration of the archive server. My ultimate goal is to get an address like “/mxcl/homebrew/tarball/master” to serve archives without any redirects. But, this first step runs stable, using less memory than a single Sinatra instance. I’m not at the point where I want to write entire applications with Node.js, but it feels like a great fit for certain cases like this.
One of the other bonuses of using Node.js: a socket.io web socket server for monitoring:
Written by
Related posts
![](https://github.blog/wp-content/uploads/2023/10/Collaboration-DarkMode-2.png?resize=400%2C212)
2024 is the biggest global election year in history. What’s at stake for developers?
GitHub is considering what is at stake for our users and platform, how we can take responsible action to support free and fair elections, and how developers contribute to resilient democratic processes.
![](https://github.blog/wp-content/uploads/2023/10/AI-DarkMode-2.png?resize=400%2C212)
GitHub named a Leader in the Gartner first-ever Magic Quadrant for AI Code Assistants
This year, as part of its annual Magic Quadrant series, Gartner published a first-of-its-kind report analyzing the state of play in the AI Code Assistants market–and named GitHub a Leader.
![AI in software development 2024 survey results with colorful graphics and percentages.](https://github.blog/wp-content/uploads/2024/08/ai-survey-header.png?resize=400%2C212)
Survey: The AI wave continues to grow on software development teams
We surveyed 2,000 people on software development teams at enterprises in the U.S., Brazil, India, and Germany about the use, experience, and expectations around generative AI tools in software development.