How the community powers GitHub Advanced Security with CodeQL queries

Security alerts produced by static application security testing (SAST) tools are valuable only if they are able to drive efficient fixes and more secure code practices without slowing developers down. The authors of “What Developers Want and Need from Program Analysis: An Empirical Study”^[1] found out that five key pain points of SAST tools for developers are: showing default checks that don’t really matter, false positives, difficulty to fit into their workflow, unclear messages, and missing suggested fixes.

The GitHub Security Lab’s CodeQL bounty program fuels GitHub Advanced Security (GHAS) with CodeQL queries written by the open source community. In this blog post, I’ll discuss how these queries solve these pain points and make GHAS security alerts more effective for developers and their teams.

See alerts that matter

The CodeQL queries that are submitted via this bounty program must demonstrate that they detect one or several past common vulnerabilities and exposures (CVEs). They must also detect the vulnerability pattern in several active open source projects. With these two requirements, we make sure that the community contributions detect real past vulnerabilities and also cover a repeatable active pattern instead of being tailored to one specific CVE.

The submissions are then triaged and evaluated by the GitHub Security Lab’s security research team. Our researchers work with the bounty hunter and propose improvements to the original submission, to extend the scope detected by the query or to focus the query on impactful results. This is to ensure that the query detects the most severe and frequent vulnerabilities.

In the last 12 months, 25 queries proposed through our bounty program detected high or critical severity CVEs such as this improper verification of cryptographic signature, detected by this community-contributed query.

Seamless integration into the SDLC

In order to support developers without slowing them down, GHAS shows security alerts as reviews within pull requests. This way, developers see these alerts the exact same way they would see a human reviewers’ comments — at the same place, at the same time. They don’t need to use another tool or view, nor do they need to add another phase to their current process.

Low false positive rate

One of the biggest problems developers face when leveraging SAST tools is false positives.

The CodeQL bounty program takes into account five equal factors to assess the bounty reward:

• Vulnerability severity
• Vulnerability scope
• False positive rate
• Code quality
• Documentation quality.

Only submissions with an acceptable false positive rate are accepted, and only the ones with a very good precision – annotated with a high or very-high precision metadata – are eventually promoted to the standard queries set, the one that runs by default with GitHub code scanning and publish alerts within the pull requests.

During the bounty assessment process, both our security researchers and the GitHub CodeQL team interact with the query author to improve the query precision. Even after this phase, the CodeQL team continues to work on these queries to bring their precision to a very high level and to promote the improved queries to the standard set.

And the improvement process is continuous: our bounty program also offers rewards for community contributions that simply improve the precision of existing queries. If a community member finds a false positive and proposes a way to remove this pattern from the query, they will also get a bounty!

Developers have autonomy to act on false positives

Even if the built-in precision is very good, it cannot be perfect. Perhaps an alert that is a true positive in most contexts is a false positive in yours. One advantage of using CodeQL is that it’s code! Developers can easily hack the query and adapt it to their context in order to permanently get rid of a false positive. Easily? Well, yes, if the code is maintainable. This is why the code quality is one of the evaluation factors of the CodeQL bounty program. During the review process, the GitHub CodeQL team ensures that the proposed query meets a high bar for requirements, which is key for the maintainability of our standard query set. And as with precision improvements, the CodeQL team will continue working on the query after submission to bring it up to the quality level necessary for you to read it and hack it.

This community-proposed query detects the CWE-090 LDAP injection pattern with a high precision and runs by default with GitHub code scanning.

Actionable messaging and suggested fixes

In order to actually shift left with secure coding practices, it’s not enough to just throw security alerts at developers. We must also empower developers to understand the problem, and to own the security requirement in the future. Application security programs usually want to take advantage of these alerts to educate the developers about secure coding best practices.

CodeQL security alerts are shipped with built-in documentation, containing an explanation of the vulnerability and its impact, remediation advice and secure coding best practices, and relevant references that you can visit. Documentation quality is another key factor that we assess during the bounty evaluation.

As I mentioned previously, developers shouldn’t have to jump to another place to access this documentation. It is shown directly within the alert, in the pull request. Developers will find the explanation and the remediation advice directly where they need it: in the same place where they will code the fix.

This query documentation explains the vulnerability and suggests a fix.

Often, an organization’s Application Security team will have their own preferred remediation advice, relevant to the organization’s context. This team can customize the query help for their developers. Instead of (or in addition to) the public references already proposed, the customized query help can point to your organization’s specific references or education material.

Conclusion

The GitHub Security Lab bounty program steers community contributions toward queries that find vulnerabilities that matter, are of high precision and high quality, and that not only alert but also educate developers. These requirements are key to integrating GHAS into your organization’s SDLC and application security programs.

[1] What Developers Want and Need from Program Analysis: An Empirical Study Maria Christakis, Christian Bird