
GitHub Availability Report: August 2023
In August, we experienced two incidents that resulted in degraded performance across GitHub services.
Last month, we announced the third anniversary of our Bug Bounty Program. While there's still time to disclose your findings through the program, we wanted to pull back the curtain…
Last month, we announced the third anniversary of our Bug Bounty Program. While there’s still time to disclose your findings through the program, we wanted to pull back the curtain and give you a glimpse into how GitHub’s Application Security team triages and runs it.
Over the three years of our program, it has evolved to streamline our internal processes and to resolve (and pay out!) the submitted issues as quickly as possible. As with most processes, we have iteratively refined and formalized the steps we take for every bounty submission received through the program. Ideally, the details of our process in this post will help other security teams looking to launch or improve their bounty process, as well as offer transparency to our researchers.
Additionally, we have released a HackerOne API client library developed for our workflow. We hope other bug bounty teams utilizing HackerOne can leverage this to add or improve the automation within their program.
When the GitHub Application Security Team launched the program in 2014, we had several key goals in mind. One particular goal was to ensure that the people taking the time to research and find vulnerabilities in our products were treated and communicated to in a way that respected the time and effort they put into the program. We have strived to maintain a knowledgable and appreciative first response to every submission received.
As the Application Security team has grown in responsibility and duties, it has been hard to manage the effort required for sufficient review and communication amongst other daily tasks. To help maintain sufficient attention to the Bounty program, each member of the Application Security team rotates daily through our on-call, First Responder, schedule. One major task during this assigned day each week is to handle incoming Bug Bounty triage.
The volume of submissions to the program has increased significantly each year (2014, 2015). Given this growth, we identified that there are major advantages of having one person handle the Bug Bounty triage per day. This allows the First Responder to focus exclusively on the Bug Bounty program, without switching between other work. If it is a particularly slow day, they can spend the time catching up on triage backlog or pushing along previous issues that may have stalled on development or other tasks. The First Responder is then the owner of the issue until it is closed out or resolved. This helps to evenly distribute the load over time, reduce duplication of effort in getting up to speed on a submission, and ensure researchers have consistent communication throughout the process.
Having a daily First Responder also helps reduce the mental duplication of work. Even if another team member isn’t on call, it’s tempting to check the incoming reports to see if there are any urgent tasks to attend to. By having one person responsible per day, this allows other members to completely check out of the Bug Bounty and focus on other work, knowing that any high-risk issues will be handled immediately. In reality, specific members may be pulled in as an subject matter expert in an area related to a submission.
Finally, by setting a schedule and committing to consistent triage, it helps us keep the inbox from growing without bounds. It has also helped us avoid periods of Bug Bounty neglect when other team priorities, such as internal code review for new feature launches, demand our full attention.
To ensure consistency throughout the team and create an easily flowing process with researchers, we have created some guidelines for the First Responder’s handling of initial triage. This is not a strict set of rules set upon the First Responder, but a workflow that we have found to be useful to best triage the submissions received. Additionally, to streamline the process for the First Responder, we utilize canned responses for a number of common submissions that typically do not make it to the further stages of triage.
The general steps taken during the initial handling of a submission are:
When we communicate our decision on the validity of the issue to a researcher, we also detail next steps in the process. These steps include the engineering team or Application Security team developing a fix based on the discussed remediation and the Application Security team determining the finalized risk of the issue.
To determine the risk used for our internal prioritization and as a mapping to our payout structure for bounty submissions, we group issues into fairly broad buckets:
We use this same rating to determine the payout of a vulnerability, as well as to express prioritization to the engineering teams. The Application Security team maps each of these risk buckets to also determine a recommended target time to fix, the urgency in which we should escalate the issue, and – if it affects GitHub Enterprise – what should the patch release cycle look like.
All vulnerabilities identified, either internally or externally through the Bug Bounty program, are handled within GitHub’s Engineering teams the same as any other bug would be. Application Security offers our recommendations around remediation and prioritization based on the determined risk. For some issues, depending on the root cause, a member of the Application Security team will fix the issue and request a review by the responsible engineering team. In other cases, such as a larger, more impactful change, the engineering team will take the lead, consulting with the Application Security team for validation of the issue. In all cases, the specific vulnerability is not only fixed, but investigation and analysis is performed to see if other similar code paths could have similar vulnerabilities.
In addition to code fixes, we strongly believe that, like with all bugs, test cases should have caught the issue before it was shipped. We work with the engineering teams to ensure proper test coverage is also included as part of the remediation work, specifically reviewing that we have negative test cases. Similarly, we have internal tooling in place to perform static analysis during the development lifecycle. We use vulnerabilities as a chance to refine and improve the analysis performed during development to catch similar issues in future development.
Depending on the timeline for the corresponding engineering work, a fix may or may not be shipped by the time we get to the fun part: rewarding our intrepid researchers for their hard work. Using the determined risk bucket, we derive a dollar amount to pay for the submission. Over the past three months, we have paid bounty hunters over $80,000 in rewards, with an average award of $1,200 per payout.
After the payout has been determined and communicated, we use HackerOne to issue the payout amount and send some GitHub Security Swag to the researcher. We then close out the report on HackerOne.
In addition to a cash payout, there are few other perks we award to our researchers. As an added bonus we apply a coupon to the researcher’s GitHub account providing 100% off unlimited private repositories for a year. For repeat researchers we extend a lifetime coupon.
We will also add researchers to the @GitHubBounty organization. If they accept the team invitation, a Security Bug Bounty Hunter
badge is added to their public GitHub profile. We use this organization to enable soon-to-be-released features to give researchers a head start on finding vulnerabilities.
When we started the program in 2014, we wanted a way to be transparent about the submissions we received and fixed. We determined that the best way to do so would be to build and maintain a GitHub Pages and Jekyll-based site at https://bounty.github.com. We use this to give an extra shoutout to our researchers and publish a quick writeup of the submissions we have paid out. We feel that this helps our users know what we are doing to fix and secure our products, gives our researchers credit for their awesome work, and hopefully helps other application security teams learn from our experiences with the program.
In running the program, we noticed that the final two steps, adding coupons and teams to a researcher’s GitHub account and writing up posts to the bounty site, were consuming a fair amount of our time. These tasks usually occurred after a fix happened and carried less urgency than the rest of the process, sometimes getting stale and forgotten due to the manual steps required.
With HackerOne’s release of an API, we took the opportunity to automate these final steps. By issuing a command in our chat system, we can open a PR to our GitHub Pages repo for the bounty site as well as apply coupons and team membership to the researcher’s GitHub account. The bounty site post is templated with the the majority of the required metadata, allowing us to get to the meaty part of the writeup without unnecessary copy and pasting. The hackerone-client
library was developed to interface this internal tooling with the HackerOne API.
GitHub’s Bug Bounty program has been evolving for the past three years and we’ve learned from the peaks and valleys it has experienced. We have seen moments of overwhelming participation that tax our resources, as well as moments of neglect as our team has shifted priorities at times. Because of these experiences, we’ve been able to create a process that allows our team to work smartly and efficiently.
As we expand the program in the future, we will continue to adapt our tools and processes to fit our needs. We would love feedback from both bug bounty researchers as well as other bug bounty teams. Send us a message on Twitter at @GitHubSecurity.
GitHub’s Application Security team is also looking to expand. If you are interested working alongside us, check out our Application Security Engineer job listing.