Scaling Lesson #23742

GitHub was created as a side project, but it seems to have struck a nerve and gained traction quickly. As such, a lot of the infrastructure decisions were made not figuring on this sort of growth:

One of the major pieces of the infrastructure is how we store the repositories. The way it was originally setup worked great for a while, but it wasn’t sustainable.

As an example, lets take my github-services repository. Here’s where it was stored prior to yesterday:

/our-shared-drive/pjhyett/github-services.git

Straight forward and simple, as well as having the added benefit of the repo being easily locatable in the file system if we needed to debug an issue.

That works well unless you have thousands of folders sitting in the same directory. GFS tried as best as it could, but with the amount of IO we do at GitHub writing to and reading from the file system, a change had to be made quickly.

After migrating last night, taking the same repository, this is where it’s currently stored:

/our-shared-drive/5/52/af/b5/pjhyett/github-services.git

Instead of every user sitting in one directory, we’ve sharded the repositories based on an MD5 of the username. A large change to be sure, but with some number crunching by our very own Tom Preston-Werner, he told me everyone on the planet can sign up twice and we still won’t have to change the way we shard our repositories after this.

Another interesting point worth mentioning is the first directory, ‘5’, was setup specifically so we could add multiple GFS mounts (we currently use just one) using a simple numbering system to help scale the data when we start bumping up against that wall again.

Now, the question you may all be asking is why we didn’t do this from the beginning. The simple answer is it would have taken more time and prevented us from launching when we did. We could have spent a couple of extra weeks in the beginning figuring out and preventing bottlenecks, but the site may not have taken off and then we would have built a scalable site that three people use.

Truth be told, it’s a great problem to have, and the site is humming along smoothly now. Now we can get back to doing fun things like building new features for you guys and gals. Keep an eye out for the big one we’re launching next week!

Written by

PJ Hyett

@pjhyett