Be aware of these top 3 risks in your CI/CD environment

In recent years, ever more organisations are becoming aware of the neccessity of CI/CD to deliver software of higher quality, faster. Since the complexity of software exploded, it became almost impossible to deliver software without a CI/CD environment.

However, a CI/CD environment opens the door to a new set of risks that can have devastating effect on your organisation, if not managed correctly. On the other side, if you do it well, these risks can be turned into great opportunities!

In this artifcle we will discuss 3 important risks in a CI/CD environment and how to mitigate them.

Software dependencies from unknown sources

Most organisations want to move fast, and that’s understandable. However, when you do not prepare for speed, you might crash horribly.

When developers create software, they make use of libraries, pieces of software they did not write themselves, but can reuse them in their own project. This is normal practice (and it is good, because this way developers do not need to reinvent the wheel everytime they start a new project), but it can have unintended consequences if the chosen libraries are downloaded from non-verified sources.

The risk

A lot of libraries are stored on platforms called registries, centrals hubs where developers can download and publish libraries. Some public registries, like NPM (Node Package Manager) are notoriously insecure: there is no formal security process and everyone is allowed to upload everything.

One way attackers try to abuse public registries is using a technique called “typo squatting”: they publish a library with a very similar name to a popular existing library. When a developer down-stream makes a typo mistake, it might download the malicious library published by the attacker instead of the original one. An example might be the NPM library socket.io, for which the attacker has uploaded his malicious versions, soket.io, soocket.io and sokket.io.

If a developer on your team wants to use this library and makes a small typo, the malicious library might end up in your build environment, and even in production, if no proper security controls are in place!

The mitigation

To prevent your organisation becoming a victim of malicious software libraries/dependencies, it is important to have a software library policy. You will also need a mechanism to enforce the policy.

Some policy ideas:

• use a whitelist of software registries from where developers are allowed to download libraries.
• for certain registries, you may scope the whitelist down to select publishers.
• allow developers to only work with crytpographically signed libraries (so that the library can be audited for authenticity)
• allow developers to only work with your organisation’s private registry, which can act as a proxy to public, verified libraries
• require a Software Bill Of Materials (SBOM) for every artifact (compiled piece of software) created in your organization (for easy and fast auditing)

OWASP : CICD-SEC-03 Dependency chain abuse

Malicious Job Agents

A typical CI/CD systems consists of a Git Server, that communicate with an arbitrary number of Job Agents. A Job Agent (or Job Runner) is a server that executes CI/CD pipelines, typically when a commit is checked in to a git repository on the Git Server. The CI/CD pipeline could be a set of instructions to automatically compile the code in the git repository.

The risks

Most CI/CD systems Git Server and Job Agents communicate via HTTPS. Authentication is mostly done using some kind of token. When the Git Server is publicly reachable (e.g. when a SaaS system is used), it is not unthinkable that an attacker finds a way to spoof a Job Agent. It could run the Agent software from an environment under his control, and request pipelines to run on the spoofed agent.

This would open up a lot of ways to steal information and work his way up in your organization. Another way an attacker could abuse your Job Agents, is by infecting the base system used for your Job Agent (which can be completely different than the base system for your Git Server). In a way this is similar to the first risk (Software dependencies), which not only applies to the code you build, but also to the code that runs the code you build :-).

The mitigation

To make sure that you don’t run pipelines on malicious Job Agents, you can take measures like:

• use a privately hosted CI/CD system
• maintain strict access controls with regards to Job Agents
• use ephemeral Job Agents

OWASP : CICD-SEC-07 Insecury system configuration

Pipeline poisoning

A CI/CD pipeline is a set of instructions that run code. This means the pipeline itself is also code. When someone with malicious intent, but no access to your IT infrastructure, can gain access to a pipeline, it can leverage this to force access to your infrastructure!

The risk

An attacker without privileged access, but with write access to a Git repository hosted on your CI/CD system, might be able to leverage this to work his way up to administrator access to your whole IT infrastructure. If your systems are not sufficiently secured, the attacker could commit malicious code that would be executed by a pipeline on a Job Agent, consequently starting a rampage.

Even without access to the pipeline file itself, the attacker could potentially stage an attack using files referenced by the pipeline. These could be build configuration files, test files or even linting tools! If the attacker is able to modify one of such files, he might be able to run malicious commands masked as a legitimate command alias.

The mitigation

To prevent someone from poisoning your pipelines, you should:

• maintain strict access controls for your CI/CD system
• limit the use of secrets in your pipelines to an absolute minimum
• use isolated environments for unreviewed code

OWASP : CICD-SEC-04 Poisoned pipeline execution

As you can see, securing a CI/CD system is a complex tasks with a unique set of attack vectors. If you worry that your CI/CD systems is exposed to one of these risks, make sure to get in touch with the team at Gluo for specialized assistance.