Applying DevSecOps to your software supply chain

This article was originally published on InfoWorld, and is republished here with permission. This is part of our blog series on DevSecOps and software security.

Developers often want to do the ‘right’ thing when it comes to security, but they don’t always know what that is. In order to help developers continue to move quickly, while achieving better security outcomes, organizations are turning to DevSecOps. DevSecOps is the mindset shift of making all parties who are part of the application development lifecycle accountable for security of the application, by continuously integrating security across your development process. In practice, this means shifting security reviews and testing left – from auditing or enforcing at deployment time, to also checking security controls earlier at build or development time.

For code your developers write, that means providing feedback on issues during the development process, so the developer doesn’t lose their flow. For dependencies your code pulls in as part of your software supply chain, what should you do?

Let’s first define a dependency. A dependency is another binary that your software needs in order to run, specified as part of your application. Using a dependency allows you to leverage the power of open source, and to pull in code for functions that aren’t a core part of your application, or where you might not be an expert. They often define your software supply chain — GitHub’s 2019 State of the Octoverse Report showed that on average, each repository has more than 200 dependencies (disclosure: I work for GitHub). An upstream vulnerability in any one of these dependencies means you’re likely affected too. The reality of the software supply chain is that you are dependent on code you didn’t write, yet the dependencies still require work from you for ongoing upkeep. So where should you get started in implementing security controls?

Create a unified CI/CD pipeline to shift security controls left

Part of the goal of DevSecOps, and shifting left, is to provide not only feedback but also consistency and repeatability as part of the development environment. This isn’t unique to your supply chain, but applies to any security control.

The earlier you can unify your CI/CD pipeline, the earlier you can implement controls, allowing your security controls to shift left. You don’t want to apply the same controls multiple times in different systems – it doesn’t scale, spreads your (already thin) security resources even thinner, allows inconsistencies to be introduced via drift or incompatibility in systems, and potentially worst of all, means you might miss something.

The precursor to shifting left and applying DevSecOps isn’t a security control at all – it’s about improving developer tooling to provide a consistent way to write, build, test, and deploy code. Introducing a centralized system for any one of these can help you improve your security. Organizations will frequently tackle developer tools from the last step, and work backwards to the first – that is, adopting a consistent deployment strategy before adopting a consistent build strategy – with one exception, code. Even if you build locally, chances are, you’re checking your code in for posterity.

You can start applying security controls to your code even without getting all the other steps unified. A developer-centric approach means your developers can stay in context and respond to issues as they code, not days later at deployment, or months later from a penetration test report. Building on a unified CI/CD pipeline, here are some tips on how your development team can apply DevSecOps to secure your software supply chain.

Declare dependencies in code, so you discover them at development time

First things first, in order to maintain your dependencies – for example, applying security patches – you need to know what your dependencies are. Seems straightforward, right?

There are many ways to detect your dependencies, at different parts of your development process: by analyzing at the dependencies declared in code, for example, specified by a developer in a manifest file or lockfile; by tracking the dependencies pulled in as part of a build process; or by examining completed build artifacts, for example once they’re in your registry. Unfortunately, there is no perfect solution as all methods have their challenges, but you should pick the solution that best integrates with your existing development pipeline or use multiple solutions to give you insights into dependencies at each step in your development process.

However, there are benefits to detecting dependencies in code, rather than later: you’re shifting that dependency management step left. This allows developers to immediately perform maintenance for dependencies – performing updates, applying security patches, or removing unnecessary dependencies – without waiting for feedback from a build or deployment step. And, if you don’t have a centralized or consistent build pipeline, and can’t apply a check later, detecting your dependencies in code means you can still infer this information. The main downside to detecting dependencies in code is that you might miss any artifacts pulled in later, for example, Gradle allows for dependencies to be resolved as part of a build, meaning build-time detection will contain more complete information.

To accurately detect dependencies in code – and to more easily control what dependencies you use – you’ll want to explicitly specify them as part of your application’s manifest file or lockfile, rather than vendoring them into a repository (forking a copy of a dependency as part of your project, aka copy-pasting it). Vendoring makes sense if you have a good reason to fork the code, for example, to modify or limit functionality for your organization, or use this as a step to review dependencies (you know, actually tracking inputs from vendors). (Some ecosystems also favor using vendoring.) However, if you’re planning on using the upstream version, vendoring makes updating your dependencies harder. By specifying your dependencies explicitly, it’s easier for your development team to update: with a single line of code change in a manifest, rather than re-forking and copying a whole repository. In certain ecosystems, you can use a lockfile to ensure consistency, so you’re using the same version in your development environment as you are for your production build, and review changes like any other code changes.

Provide a path paved with ‘golden’ packages for your development team

You might already be familiar with the concept of ‘golden’ images, which are maintained and sanctioned by your organization, including the latest security patches. This is a common concept for containers, to provide developers with a base image on which they can build their containers, without having to worry about the underlying OS. The idea here is to only have to maintain one set of OSes, managed by a central team, that you know have been reviewed for security issues and validated in your environment. Well, why not do that for any other artifacts too?

To supplement a unified CI/CD pipeline, you can provide a reference set of maintained artifacts and libraries. This is just a pre-emptive security control – rather than verifying that a package is up to date once it’s been built, give your developers what they need as an input to their build. For example, if multiple teams are using OpenSSL, you shouldn’t need every team to update it; if one team updates it (and there are sufficient tests in place!), then you should be able to change the default for all teams. This could be implemented by having a central internal package registry of your known good artifacts, that have already passed any security requirements, and have a clear owner responsible for updating if new versions are released.

By providing a single set of packages, you’re ensuring all teams reference these. Keep in mind, the latest you can do this is in the build system, but this could also be done earlier in code, especially if you’re using a monorepo. An added benefit of sharing common artifacts and libraries is making it easier to tell if you’re ‘affected’ by a newly discovered vulnerability. If the corresponding artifact hasn’t been updated, you are! And then it’s just one change to address the issue, and for the update to flow downstream to all teams. Phew.

Automate downstream build and deployment

To make sure that developers’ hard work pays off, their changes actually need to make it to production! In creating a unified CI/CD pipeline, you cleared a path for changes that are made to code in a development environment to propagate downstream to testing and production environments. The next step is to simplify this with automation. In an ideal world, your development team only makes changes to a development environment, with any changes to that environment automatically pushed to testing, validated, and automatically rolled out (and back, if needed).

Rather than applying DevOps and DevSecOps by requiring your development team to learn operations tools, you simplify those tools and feedback to what these teams need to know in order to make changes where they’re most familiar, in code. This should sound familiar – it’s what’s happening with trends like infrastructure as code, or GitOps – define things in code, and let your workflow tools handle making the actual change.

If you can automate downstream build, testing, and deployment of your code – then your developers only need to focus on fixing code. Following DevSecOps principles, they don’t need to learn tooling to do validating testing, phased deployments, or whatever you might need in your environment. Crucially, for security, your development team doesn’t need to learn how to roll out a fix in order to apply a fix. Fixing a security issue in code and committing it is sufficient to ensure it (eventually) gets fixed in production. Instead, you can focus on quickly finding and fixing bugs in code.

Creating a unified CI/CD pipeline allows you to shift security controls left, including for supply chain security. Then, to best apply DevSecOps principles to improve the security of your dependencies, you should ask your developers to declare your dependencies in code and in turn provide them with maintained ‘golden’ artifacts and automated downstream actions so they can focus on code. Since this requires changes to not just security controls, but your developers’ experience, just using security tooling isn’t sufficient to implement DevSecOps. In addition to enabling platform-native dependency management features, you’ll also want to take a closer look at your CI/CD pipeline and artifact management.

Altogether, applying DevSecOps means you can have a better understanding of what’s in your supply chain. By using DevSecOps, it should be simpler to manage your dependencies, with a change to a manifest or lockfile easily updating a single artifact in use across multiple teams, and automation of your CI/CD pipeline ensuring that changes developers make quickly end up in production.

Written by

Maya Kaczorowski

@mayakacz