An analysis on developer-security researcher interactions in the vulnerability disclosure process
We put out a call to open source developers and security researchers to talk about the security vulnerability disclosure process. Here’s what we found.
This blog post is a special report providing insights into developers’ interactions with security researchers through the vulnerability disclosure process and their views and perspectives on the security research community. The analysis is brought to you from the GitHub Security Lab.
In December 2020, we announced that we were expanding our research to understand more about the relationship between the developer and security research communities. Developers and security researchers have sometimes had a fraught relationship. To better understand these dynamics, we honed in our research on the vulnerability disclosure process, during which both communities are expected to interact to address potential threats.
Currently, there is no standard vulnerability disclosure process for the broader security research community. Instead, entities such as Google’s Project Zero, the GitHub Security Lab, Snyk, HackerOne, and VuSec have their own processes for disclosing vulnerabilities. In a system that lacks uniformity, understanding the relationship among stakeholders can lead to more effective actions to remedy vulnerabilities, ultimately leading to a safer, secure, and more collaborative ecosystem.
We put out a call to open source developers and security researchers to talk about the security vulnerability disclosure process. We’ve leveraged some findings to improve the lab’s process and guide additional research topics. We also believe our findings are relevant to the wider community and want to share some highlights.
This report focuses on maintainers’ perspectives, and we plan to expand our analysis later this year to include insights from the security research community.
Table of contents
Summary
GitHub Security Lab conducted qualitative surveys of open source maintainers via remote interview sessions between November 2020 and March 2021. See Appendix: research methodology for full details. The findings reported here are split across three categories:
Dynamics between maintainers and security researchers: The maintainers who participated in our study had little to no engagement with the security research community beyond the vulnerability disclosure process. While the interactions that occurred could be positive, many maintainers didn’t find the security research community particularly welcoming. Despite this, maintainers are open to learning foundational information about security research.
| “My initial emotional response is “Oh no, how bad is it now?” It’s never fun being on the receiving end. It depends on the severity of the impact. … Should I drop my professional work to fix it? If [it’s] critical enough, I sometimes do that.” |
Maintainers’ communication preferences: Upon receiving vulnerability reports, maintainers typically experience a range of emotions, including anxiety and stress; however, they are appreciative of constructive feedback that is communicated to them through private channels.
Maintainers’ perceptions of the vulnerability reporting process: To determine the severity of the reports they receive, maintainers reported assessing impact first. They did not expect security researchers to provide remediation advice but were appreciative when it was offered. Participants typically felt capable of addressing potential vulnerabilities within the commonly used 90-day coordinated disclosure remediation window.
A detailed look at our findings
What’s the dynamic between maintainers and security researchers?
We investigated how developers and security researchers interact during the vulnerability disclosure process to surface ways of improving these encounters and forming stronger relationships.
Maintainers’ level of engagement with the security community
The majority of participants explained that they have had little to no engagement with the security research community.
- Beyond vulnerability reports, some maintainers mentioned they engage with security researchers through other avenues, like Twitter, Slack, Discord, and conversations with friends and interpersonal connections in the security space.
- Other participants characterized their engagement with the security research community as passive or as on the receiving end of information.
When it comes to resources provided by security teams, participants would like to see foundational information or “ramp up content” that relates to:
- common classes of vulnerabilities and common issues/vulnerability patterns,
- maintainer expectations (e.g., open source projects take longer to respond due to a lack of resources),
guides for security researchers, and - coverage on topics, including workflows, how security works, and how bugs are found.
Interactions with security researchers
Maintainers recognized tension among stakeholders (engineers, designers, and security researchers) with differing priorities. While maintainers reported that their interactions with security researchers were generally positive or pleasant, this wasn’t universally true. Some characterized their interactions with security researchers as “very obscure” and “it runs the gamut,” depending on the researcher.
“My assumption is that in any software stack there is a tension between engineers and designers and security. Everyone has different priorities. … It creates conflict. If everybody understands each other’s priorities, [then] usually you can make decisions and understand trade offs. Everyone is at least less unhappy about the result. Security doesn’t work unless everyone is on board [and] aligned with those priorities. Everyone involved needs to understand that security is important.”
“Pretty positive, mostly because I know what their perspective is. We haven’t had terrible incidents with anyone yet as the maintainer.”
What are maintainers’ communication preferences?
To step toward a more effective process, it’s important to understand the impact of receiving vulnerability reports on maintainers and how they perceive this communication. We hope these insights lead security researchers to gain a better understanding of maintainers’ communication styles and preferences to allow for a more collaborative partnership.
Preferences for receiving feedback
Generally, maintainers welcome constructive criticism. If the feedback is overall actionable and widely applicable, maintainers are willing to accept it even if there is a negative aspect to it. If feedback is a negative, personal attack, maintainers tend to ignore it or set boundaries by communicating what type of discussion they will engage in and what they will not.
Notification methods for security risks
Some maintainers said they want to receive notifications similar to how the GitHub Security Lab sends its notifications. Through this process, where a researcher identifies a vulnerability in a project and details (in a report) a summary of the problem, an explanation of the specific issue, the vulnerability’s potential impact, and remediation advice. Maintainers identified various ways in which they receive security notifications, including email notifications, Twitter direct messages, and discussions through Spectrum chat. While some maintainers may receive notifications via email, others found emails “overwhelming” and would prefer other forms of notifications, particularly push notifications.
“We don’t have a specific email domain that can route emails to the maintainer efficiently, and we were hesitant to put our personal emails directly in a markdown file from a privacy perspective.”
One maintainer mentioned hesitancy in publishing a security policy due to reluctance to publish their email address.
“Wishlist: have a way to reach out privately, without needing an out-of-band email.”
Another maintainer recounted one instance when a possible vulnerability was reported via a public issue.
“In the past, [a public issue] has been the primary way that people opened another topic or something on the forum … and then claimed they had found something, with a corresponding triumphant ‘lol’.”
Actioning a report
Once maintainers receive a security notification, they explain feeling an initial range of emotions ranging from appreciation and happiness to stress and alarm, but ultimately take action:
“My initial emotional response is “Oh no, how bad is it now?” It’s never fun being on the receiving end. It depends on the severity of the impact. … Should I drop my professional work to fix it? If [it’s] critical enough, I sometimes do that.”
“The majority of [my] users will update to the safe version before the CVE goes out, which means most of my users will never even know I had a vulnerability unless they care to look. And if they care to look, they’ll see that I was a good maintainer. And I actually fixed it right away.”
Some maintainers said they want to receive notifications which detail (in a report) a summary of the problem, an explanation of the specific issue, the vulnerability’s potential impact, and remediation advice. This aligns with how GitHub Security Lab delivers reports.
How do maintainers perceive and approach the vulnerability reporting process?
It is important to further explore what type of information maintainers prioritize to understand the severity of a submitted report as well their views on remediation advice and the ability to address the report within the commonly used 90-day timeframe.
- Overwhelmingly, maintainers prefer that reports be submitted to them privately.
- When maintainers first look at a report to identify its severity, they typically assess impact to determine whether there is Remote Code Execution (RCE), which per one participant “screams high severity.” In addition, maintainers look at code snippets and reproducible steps.
- A majority of maintainers have a designated security contact via email. However, most do not yet have a security policy and either want to implement one or are actively working on creating one.
- When it comes to providing remediation, maintainers do not necessarily see it as a requirement for security researchers to provide but would find it helpful if they did so, since open source maintainers volunteer their time.
- Regarding the common 90-day disclosure deadline, most maintainers said they think that this is a reasonable, fine, or fair timeline to fix an issue. While there is a general consensus that 90 days is a standard, reasonable timeline, maintainers also want flexibility with this timeline and a more collaborative process in terms of determining the timeline.
Next steps for GitHub Security Lab
- Encourage increased partnership. A strong partnership between open source developers and security researchers is tantamount to securing open source software. The Security Lab will continue to investigate effective methods to foster collaboration between these communities.
- Provide foundational information. The Security Lab already strives to share its research with developer communities in a relatable way. We will continue to build on this work and empower developers with educational content with a security focus.
- Continue exploration. We’ll be talking to security researchers about their experiences interacting with maintainers in the vulnerability disclosure process. This will help us gain a fuller understanding of stakeholders’ experiences in the process. Additionally, we asked preliminary questions during this study about how security is depicted through visuals and potential alternative imagery. We will further explore alternative descriptors and language that the Security Lab and the broader security research community can leverage as a way to attract developers.
We welcome your feedback as we continue to explore ways to foster effective partnership between the security research community and open source maintainers. If you’re a data nerd and want to see the full details of our methodology and research, check out the appendix below, or follow us on Twitter to stay up-to-date on our latest security research.
Appendix: research methodology
Between November 2020 and March 2021, we interviewed nine open source maintainers across large and small open source projects.
Participant recruitment
GitHub Security Lab’s goal was to explore open source maintainers’ experiences with the security research community. We recruited experienced open source software (OSS) maintainers, with experience ranging from approximately four years to 20 years.
We leveraged several recruitment strategies to identify participants. We reached out to open source maintainers with whom the Security Lab’s security researchers had previously engaged. To supplement, we turned to other GitHub programs, including GitHub’s Top 100 Maintainers Group, GitHub’s Early Access Group, and GitHub Stars. We also published an open call via blog post. We offered an interview incentive of a $150 credit to go towards each interviewee’s nonprofit organization of choice.
It was challenging to recruit open source developers to discuss security topics, as maintainers expressed not being well-versed in security topics or with the vulnerability disclosure process. Therefore, as part of our recruitment process, we used snowball sampling by asking interviewees if they had recommendations for other participants. We recognize that our recruitment experience may be unique to this work and welcome other perspectives, which may differ from our experience.
Interview data collection
We conducted nine, one-hour virtual interview sessions with each participant during which we posed 13 multi-part questions. These questions were intended to assess open source maintainers’ interactions with the security research community broadly as well as to receive specific feedback from participants on the effectiveness of the Security Lab’s current vulnerability disclosure process.
We conducted the interviews in three main sections: (1) introductory questions about participants’ experiences as open source maintainers, (2) their interactions with security researchers and their views towards the community, and (3) specific questions about the Security Lab’s vulnerability disclosure template and process. To conclude the interviews, we asked additional questions about general industry practices and allowed time for open discussion for interviewees to provide additional feedback and ask questions of the interviewers.
Data analysis
We qualitatively analyzed interview responses and supplemented our understanding of responses with interview transcripts. Then, we coded each response and identified specific themes related to each interview question.
Tags:
Written by
Related posts
Attacking browser extensions
Learn about browser extension security and secure your extensions with the help of CodeQL.
Cybersecurity spotlight on bug bounty researcher @adrianoapj
As we wrap up Cybersecurity Awareness Month, the GitHub Bug Bounty team is excited to feature another spotlight on a talented security researcher who participates in the GitHub Security Bug Bounty Program—@adrianoapj!
Securing the open source supply chain: The essential role of CVEs
Vulnerability data has grown in volume and complexity over the past decade, but open source and programs like the Github Security Lab have helped supply chain security keep pace.