How DevSecOps Can Help Mitigate the Next Log4j Vulnerability

This post will show you how adopting DevSecOps practices can mitigate vulnerabilities in libraries like Log4j. You will find answers to questions such as:

• Why do we need logging frameworks like Log4j?
• How did a lookup plugin in Log4j impact thousands of downstream applications and devices?
• Why are responsive actions like tactical patching not enough to reduce risk when vulnerabilities like Log4j are unknown?
• How adopting DevSecOps practices, including network segmentation, testing for known vulnerabilities, and implementing artifact and code signing, can help organizations reduce the impact of supply chain vulnerabilities such as those introduced by Log4j?

What is Log4j?

When building complex applications, development teams often employ a microservice architecture pattern to achieve a maintainable, testable, and loosely coupled service structure. This allows them to collaborate, simultaneously work on and deploy components independently using a standard API to facilitate communication and operations across multiple services.

While most applications employ a certain level of log and trace capability, standardization and distributed tracing are especially important in a system of decoupled services. This helps explore how a particular request is handled to aid application analysis or debugging.

As the need for a standardized method of achieving logging and tracing grew, multiple projects were created, enhanced, and eventually refined, leading to the development of the open-source Apache Log4j project.

Here’s a great bite-sized demonstration of implementing Log4j. To further simplify, we can break this process into four steps:

1. Declare the dependency
2. Configure the Log4j properties file
3. Call the module into your source code
4. Create and use the logger in your code

A Vulnerability Emerges

Several lookup plugins were added to the project to extend Log4j capabilities with the 2.0-beta9 release, including JNDILookup.

“Currently, Lookup plugins [1] don’t support JNDI resources. It would be really convenient to support JNDI resource lookup in the configuration…”

Unfortunately, this particular lookup and enrichment plugin contained a vulnerability due to the absence of user-controlled input sanitation as outlined in CVE-2021-44228. 

“parameters do not protect against attacker-controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.”

The initial impact allowed attackers to import and execute their own malicious code into any application that used vulnerable Log4j dependencies. This resulted in some attackers achieving full remote code execution in environments lacking segmentation. Palo Alto networks’ Unit42 blog post does an excellent job of explaining the impact and exploitation of this vulnerability in detail. CISA.gov also provided notable guidance around remediation and tracking vulnerable software versions, which totaled nearly 3,000 various software packages and versions at the time of writing this blog. 

DevSecOps as a Defense Strategy

Since the emergence of the vulnerability, there has been a lot of tactical guidance to help organizations implement patching, micro patching, or establishing mitigating controls. However, this does little to address the fact that the vulnerability existed unnoticed for years and could have been silently exploited.

Below, we examine three key DevSecOps controls that can prevent exploitation, identify vulnerable dependency usage, and guide you to maintain an inventory of signed code and artifacts. 

1. Network Segmentation

The core of any good segmentation plan is identifying and documenting all of the authorized data flows the application is allowed to make. This allows you to establish a data flow diagram approach to threat modeling. With this information, you can establish barriers to prevent risky communication flows. 

For example, consider a typical 3-tier web application architecture — a public-facing web layer (typically via a proxy, WAF, or CDN), a backend application layer that initiates backend processing, and a storage or database layer that provides data or object storage. 

Assuming each layer is segmented and can only talk to the layer where authorized data flows have been identified (i.e., database can talk to backend app, but not the wider internet or other internal servers), we can mitigate a successful SQL injection vulnerability from being used against our infrastructure. 

This prevents attackers from establishing a beachhead, pivot point, or direct command and control of the application. In this scenario, SQL injection is not good but preventing a single exploited vulnerability from turning into a chain prevents a bigger catastrophe.

Using Infrastructure-as-Code as part of your DevOps toolsets allows for manual code reviews and automated scans of the codebase to identify poorly configured segmentation, preventing these configurations from reaching production. 

In the case of Log4j, if we know the Java daemon shouldn’t be making outbound calls or connections, we can disrupt the attacker’s kill chain. We can also build detection and response signatures from instrumentation that trigger a security operations investigation when an application starts behaving unexpectedly and creates a sudden influx of firewall deny events. Using an IaC approach in source control helps the incident responders track down configuration changes when and where they occur, helping mitigate the issue faster. 

2. Test for Known Vulnerabilities

There are multiple approaches to identifying vulnerabilities for applications that are within the span of your organization’s control. For commercial off-the-shelf software (COTS), seek to implement a traditional vulnerability management program. This is typically done by:  

• Developing a plan in coordination with system owners
• Implementing a vulnerability scanning solution against known applications and networks
• Assessing, triaging, and assigning remediation activity to the appropriate system or data owner
• Tracking progress and providing metrics to leadership. Such metrics should include the current risk posture and the time interval between vulnerability identification and remediation.

Integrate security testing directly into your development pipeline for applications where your team creates and maintains a codebase. Controls such as linting tools, SAST tools, or DAST tools can be configured to identify and correct code with obvious vulnerabilities or dependency calls as part of your build pipeline, preventing them from being deployed into production.

For newly discovered vulnerabilities such as those recently found in Log4j, you can use tools such as Anchore’s Syft to establish a software build of materials (SBOM), then cross-compare that output using a vulnerability scanner such as grype. This process can be scheduled and fully automated so that your development and security teams have the visibility to remediate the issue as soon as it is identified.

3. The Signing of Artifacts & Code

Finally, there are cases where an attacker may have gained access directly to your source code control system (or code repository) or that of a module that supports your application. Such was the case in the late 2020 Solar Winds compromise. 

It’s important to establish artifact flow integrity to defend against such software supply chain attacks. You can achieve this by leveraging the in-toto framework using cryptographic signing, where building steps are defined in a layout and then signed by a project owner using public-key cryptography. If an unauthorized party attempts to introduce code to a build, the validation process fails. In-toto offers a demo project to help organizations become familiar with key concepts to ensure build integrity. However, in-toto is already being used in declarative pipelines in major build CI/CD platforms such as Jenkins.

Conclusion

The Log4j remote code execution vulnerability is a symptom of a larger class of vulnerabilities that are now being identified and exploited. This is partly due to the assumed trust organizations have in open-source software. It’s also due to an inherited trust that organizations commonly extend to patches and software revisions provided by development teams.

DevSecOps processes help reduce the risk of such vulnerabilities and increase trust in your software and supply chain. They can be adopted to prevent vulnerabilities from being exploited, detect exploitation attempts, and prevent unauthorized modifications to your codebase from being deployed in a build.