Every engineering team faces the challenge of delivering new features without compromising the product’s security. The modernization of DevOps processes to include security best practices has brought this goal within every engineer’s reach.
Do you build a CI/CD pipeline with open source tools? Invest in commercial products? Take a hybrid approach? Or finally, shift-left?
In recent years, GitHub has been at the forefront of the shift-left approach, i.e., moving security tooling and processes closer to the engineering teams. Decentralizing the security process allows developers to catch bugs quickly and prevent them from making it to production.
Today, we’ll share the example of our client who successfully adopted DevSecOps practices to augment their existing GitHub Actions workflows and source code management processes.
Understanding the Problem
Modus Create had earlier performed a security assessment for the client, which strengthened their overall security posture and resulted in cost savings of $4000 a day.
Now, the client wanted to review its DevSecOps pipeline, identify gaps, and take steps to remediate potential security issues. It wished to reduce the time spent on costly re-releases and minimize the threat of cyberattacks.
Our team conducted the assessment and, after reviewing several options, decided to secure CI/CD pipelines with GitHub Advanced Security. There were several reasons behind our recommendation:
- The client’s engineering teams were heavily invested in GitHub Enterprise and had been actively migrating projects into it.
- The client was building out existing CI/CD pipelines in GitHub Actions, so some of the team members were already familiar with using GitHub to drive DevOps processes.
- GHAS provides best-in-class feature set for injecting security into the CI/CD process, with features such as secrets scanning and SAST scans across repositories and dependency scans that could identify vulnerable packages.
Solutioning with GitHub Advanced Security (GHAS)
Coupled with GitHub Actions, we decided to reduce the spread of tools, remove bottlenecks in CI/CD processes around security testing, and provide a single integrated pane of glass for DevOps, Security, and Source Control.
1. Setting up Teams
The first step was to determine the team structure. Admin users of repositories have the ability to view the Security Overview dashboard. So, it made sense for each product team to have a subset of GitHub users with the ability to gain an overview of the security issues in relevant repositories. This would help them measure the overall product health.
Therefore, we settled on the following team and sub-team model:
- Product Team
- Engineering Team
- Admin Team
- Release Engineering Team
We assigned the Maintainer role to engineers so they could read and write to the repository. The Admin team was a subset of the engineers who were team leads and had admin access. This would grant them access to review the security overview.
Finally, to support the separation of duties, we included a Release Engineering Team, which had permission to deploy releases to production or update production infrastructure. We supported locking down the release processes through branch protection rules and the CODEOWNERS files. This allowed the client to meet compliance requirements around the separation of duties. The nested team structure would also allow them to add other sub-teams, such as QA, with a different permission set.
2. Planning the Rollout
After deciding the team structure, we planned the rollout process, i.e., figuring out in what order should we onboard teams to use GHAS. The pre-designed onboarding process included a walkthrough of GHAS features and dashboards, as well as how to write, update and deploy the CodeQL file. We complemented this plan with a video guide to GHAS that teams could refer back to.
With the order of the teams decided and the script for training in place, the next step was to define the triage process for results.
3. Defining a Triage Process
Each project now had key individuals who could update CodeQL findings and decide if they fall into the following categories:
- False positives that can be ignored
- Minor or Medium issues that shouldn’t block the PR and can be converted into backlog items to be fixed when time permits
- Major issues that should be addressed and the PR re-submitted
The team baked this process into the existing SDLC to ensure that Minor or Medium findings could be addressed in the standard Sprint backlog. They also put a similar process in place for dependabot alerts. This ensured that any out-of-date dependencies are identified, upgraded, and tested as part of the standard sprint cycle.
Each company will have its own security requirements around what they should fix immediately versus fix later. This will largely depend on the type of product and the management’s risk appetite. Therefore, when configuring this gating mechanism, you should consider all the variables.
4. Enabling GHAS
With the team structure applied, training prepped, and a triage process in place, we enabled GHAS and supporting tools. This involved switching on the features at the organizational level.
When enabling these options, we also ensured that the team selected checkboxes for automatically enabling them on new repositories.
A benefit of the Secret scanning feature is that you can add custom patterns. This allows engineering teams to improve the secrets scanning process over time by tailoring it to their needs.
5. Adding the Configuration to Repositories
At the heart of enabling GHAS is the CodeQL file which is included in the .github directory of your project. With this in place, you can kick off the Static Analysis (SAST) scans.
We worked with the code base owners to add the CodeQL files to all the existing repositories for the first team. In addition, we also updated the template repositories. The template repositories act as a central location for base CodeQL configuration and ensure all future repositories built from them include CodeQL.
The team tweaked the CodeQL files to ensure they were scanning the correct languages and referencing the correct branch.
6. Branch and Pull Request Configuration
As a final step, the team updated the branch configuration to ensure that issues identified by CodeQL as Major failed a pull request. All pull requests also required at least one reviewer before you can merge into lower environment branches. The production (main) branch required two pull requests, with one coming from an individual in a team listed in the CODEOWNERS file (in this case the Release Engineering Team).
Benefits of Shifting Left with GHAS
Implementing GHAS injected security into the client’s CI/CD process and brought several new capabilities such as:
- Secrets scanning across repositories by default, incorporating features such as detecting secrets from key cloud vendors, e.g. AWS and Azure, and also the ability to add in custom secrets identifiers
- Static Analysis Security Testing (SAST) scans across repositories for the key languages the company had adopted. The client can configure these scans on pull requests, as well as via cron jobs, to perform periodic scans of the mainline of development
- Dependency scans that could identify vulnerable and out-of-date packages that may increase the threat surface of the application.
These capabilities strengthened the client’s security posture, allowing them to detect and remediate security issues earlier in the process, preventing expensive re-releases and hotfixes. Furthermore, they also reduce the risk of exploitation from bad actors, preventing reputational and financial damages.
This story was a high-level overview of how we set up GHAS for a customer and integrated it into their existing DevOps processes. Each environment is different, so while this may work for one engineering team, tweaks and changes would likely be needed for your team. Reach out to Modus Create if you need help adopting DevSecOps best practices.
