GitHub Availability Report: October 2024
In October, we experienced one incident that resulted in degraded performance across GitHub services.
In October, we experienced one incident that resulted in degraded performance across GitHub services.
October 11 05:59 UTC (lasting 19 hours and 12 minutes)
On October 11, 2024, starting at 05:59 UTC, the DNS infrastructure in one of our sites started failing to resolve lookups following a database migration. Attempts to recover the database led to cascading failures that impacted the DNS systems for that site. While the team worked to restore the infrastructure, the first customer impact started around 17:31 UTC.
The impact of the incident was broad. 4% of Copilot users saw degradation in IDE code completions, while 25% of Actions workflow users experienced delays greater than 5 minutes. 100% of the code search requests failed for ~4 hour window.
Starting at 18:05 UTC, we attempted to resolve the issue by repointing the degraded DNS site to a different site but were unsuccessful. While this mitigation was effective at restoring connectivity within the site, it caused issues with connectivity from healthy sites back to the degraded site, so we started planning a different remediation effort.
At 20:52 UTC, the team finalized a remediation plan and began the next phase of mitigation by deploying temporary DNS resolution capabilities to the degraded site. At 21:46 UTC, DNS resolution in the degraded site began to recover and was fully healthy at 22:16 UTC. Lingering issues with code search were resolved at 01:11 UTC on October 12.
The team also continued to restore the original functionality within the site after public service functionality was restored. GitHub is working to harden our resiliency and automation processes around this infrastructure so we can diagnose and resolve issues like this faster in the future.
Please follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the GitHub Engineering Blog.
Tags:
Written by
Related posts
The top 10 gifts for the developer in your life
Whether you’re hunting for the perfect gift for your significant other, the colleague you drew in the office gift exchange, or maybe (just maybe) even for yourself, we’ve got you covered with our top 10 gifts that any developer would love.
Congratulations to the winners of the 2024 Gaady Awards
The Gaady Awards are like the Emmy Awards for the field of digital accessibility. And, just like the Emmys, the Gaadys are a reason to celebrate! On November 21, GitHub was honored to roll out the red carpet for the accessibility community at our San Francisco headquarters.
Students: Start building your skills with the GitHub Foundations certification
The GitHub Foundations Certification exam fee is now waived for all students verified through GitHub Education.