This Morning’s Git Protocol Outage
Anonymous Git clones were offline this morning for several hours (SSH and HTTP clones remained functional). Several days ago we removed the client inactivity timeout from our backend HAProxy configuration…
Anonymous Git clones were offline this morning for several hours (SSH and HTTP clones remained functional). Several days ago we removed the client inactivity timeout from our backend HAProxy configuration to prevent long-running RPC calls from incorrectly being severed. It turns out that some Git connections will behave badly and sit idle keeping a connection open without any data transfer. Without the timeout in place these connections eventually consume every spot within HAProxy’s session limit, meaning that any subsequent connections will be dropped. We have reinstated the timeouts and will carefully review any changes we need to make here to ensure this does not happen again.
At the same time, our monitoring of anonymous Git clones is lacking, so we were not notified of the problem as soon as we should have been. I’m now in the process of putting in place a more robust monitoring solution for the Git protocol so that in case there are problems in the future we will be notified in a timely fashion.
I’m very sorry about the problems this morning. We take all matters of availability very seriously and will do everything we can to make sure these problems do not arise again.
Written by
Related posts
![](https://github.blog/wp-content/uploads/2023/09/screencapture-innovationgraph-github-2023-09-20-15_44_54-1.png?resize=400%2C212)
How researchers are using GitHub Innovation Graph data to estimate the impact of ChatGPT
An interview with economic researchers who are applying causal inference techniques to analyze the effect of generative AI tools on software development activity.
![](https://github.blog/wp-content/uploads/2024/01/Enterprise-DarkMode-1.png?resize=400%2C212)
GitHub Availability Report: June 2024
In June, we experienced two incidents that resulted in degraded performance across GitHub services.
![](https://github.blog/wp-content/uploads/2024/06/AI-DarkMode-4.png?resize=400%2C212)
Advancing responsible practices for open source AI
Outcomes from the Partnership on AI and GitHub workshop.