State of the Hub: Rackspace day 2
Monday has come and gone, no one is happier about this than I am. As day 2 is wrapping up, I’d like to post an update for those of you…
Monday has come and gone, no one is happier about this than I am. As day 2 is wrapping up, I’d like to post an update for those of you that have been following us.
Before I get to the details, I’d like to remind everyone to open tickets on Tender if you have any issues. For those of you that have open tickets never fear, we’re burning through tickets as fast as we can. When I logged in this morning we had over 120 tickets in the inbox, right now we’re sitting at 35. Unfortunately Tender doesn’t give me stats on the number of tickets opened today, but we’ve dealt with at least 85 plus however many new tickets were opened during the day. I’d bet that’s a few dozen at least. To put that in perspective, before the move on any given stable day we saw maybe a dozen tickets, and on unstable days anywhere from 20 to 40 new tickets on average.
Bugs squashed today
Everyone was busy all day today killing bugs. At one point I estimate mojombo was handling up to 800 problems a minute. Here’s a rundown of the major bug fixes:
-
502 Bad Gateway
errors should, for the most part, be gone. These were hitting gist creation, applying commits in the fork queue and user creation the worst. If you run into this error please open a ticket and detail what you were doing when you encountered it. -
Repo under migration
errors should also be gone. This included a patch to ensure user paths were generated if they didn’t exist on lookup and a few batch jobs to force repos that had not been created on disk to generate. If you have any new, unpushed repos throwing errors please let us know. - Errors when pushing repos with submodules should be gone now.
- A well placed kick to the mail server got outgoing emails working again.
- User deletion should work again, I have not personally tested this one because I can’t bear to see users go.
- New Pages were not being set up, existing pages were not building on push, and pygments was missing on the Pages server. All of these should be fixed now. If your page has not built, make a new commit to trigger a new build. If you new page hasn’t been set up on the server, open a ticket so we can force it to run.
Outstanding issues
- Gist forking still broken.
- Zip/tar downloads are failing to generate for a few repos. Please open a ticket and let us know what repo so we can pin this one down.
- Connections to ssh.github.com:443 are still not working. We’ve already applied one fix here but it did not work. We are still working on this, firewalled folks, all I can ask is that you be patient.
- Some gists made shortly before the lockdown on Sunday did not sync to rackspace.
- Gem building is still disabled. We have no ETA on this one, but we’re not afraid to shamelessly plug gemcutter in the mean time.
and now for something completely different
Since the move, we’ve seen over 1000 new users and 2000 new repos. We’ve processed over 800k background jobs, and the background job queues are blazing fast on the new servers. Before the move our low-priority queue (network graph updates, http cloning updates and some other jobs) was backed up for a few weeks. At peak times it would rise to 40k jobs or more. Graph updates were often delayed many hours. We had 25 job runners working nearly full time on the low prio queue. On the new servers we have 40 workers, however they are sitting idle most of the time. Usually only 2 or 3 are active at any given time. The jobs queues, including the low prio queue, have stayed near zero since we cut over the DNS.
Written by
Related posts
GitHub Availability Report: November 2024
In November, we experienced one incident that resulted in degraded performance across GitHub services.
The top 10 gifts for the developer in your life
Whether you’re hunting for the perfect gift for your significant other, the colleague you drew in the office gift exchange, or maybe (just maybe) even for yourself, we’ve got you covered with our top 10 gifts that any developer would love.
Congratulations to the winners of the 2024 Gaady Awards
The Gaady Awards are like the Emmy Awards for the field of digital accessibility. And, just like the Emmys, the Gaadys are a reason to celebrate! On November 21, GitHub was honored to roll out the red carpet for the accessibility community at our San Francisco headquarters.