This Morning’s Git Protocol Outage
Anonymous Git clones were offline this morning for several hours (SSH and HTTP clones remained functional). Several days ago we removed the client inactivity timeout from our backend HAProxy configuration…
Anonymous Git clones were offline this morning for several hours (SSH and HTTP clones remained functional). Several days ago we removed the client inactivity timeout from our backend HAProxy configuration to prevent long-running RPC calls from incorrectly being severed. It turns out that some Git connections will behave badly and sit idle keeping a connection open without any data transfer. Without the timeout in place these connections eventually consume every spot within HAProxy’s session limit, meaning that any subsequent connections will be dropped. We have reinstated the timeouts and will carefully review any changes we need to make here to ensure this does not happen again.
At the same time, our monitoring of anonymous Git clones is lacking, so we were not notified of the problem as soon as we should have been. I’m now in the process of putting in place a more robust monitoring solution for the Git protocol so that in case there are problems in the future we will be notified in a timely fashion.
I’m very sorry about the problems this morning. We take all matters of availability very seriously and will do everything we can to make sure these problems do not arise again.
Written by
Related posts
GitHub for Nonprofits: Drive social impact one commit at a time
Now, it’s easier than ever for nonprofits to receive GitHub product discounts.
Career growth, learning, and fun, oh my! Your guide to GitHub Universe 2024
Dive into a whirlwind of insights, inspiring sessions, and mingle with industry leaders at this year’s Universe. Get your tickets now before they’re gone!
GitHub Availability Report: September 2024
In September, we experienced three incidents that resulted in degraded performance across GitHub services.