How to Set up Trivy Scanner in GitLab CI: The Complete Guide

Containerization is one of the modern practices being used increasingly by software development teams as the DevOps culture continues to grow in popularity. Most of these environments benefit from the rich features provided by containerization such as scalability, portability, and process isolation.

However, it is essential to consider "how secure" a software is before shipping it to your clients. When creating container images as your releases, the heavy use of third-party and outdated libraries means you run the risk of introducing added vulnerabilities to the images you ship. As such, there is a need for a reliable way of scanning container images. This is where Trivy comes in handy.

‍

What is Trivy?
Is Trivy free?
How to integrate Trivy into an existing GitLab CI pipeline
Bonus tip - scan images of a Kubernetes resource with Trivy
Conclusion

‍

What is Trivy?

Trivy is an easy-to-use, fast, and comprehensive open-source tool used by DevOps and security teams for vulnerability and infrastructure as code (IaC) scanning of containers and artifacts. Maintained by Aquasecurity, Trivy:

Works with containers, file systems, or even git repositories.
Is easy to install with no prerequisites, such as the installation of a database.
Is fast to run because there is no database involved.
Fits the DevSecOps methodology as it can be integrated into CI systems (Circle CI, Jenkins, GitLab CI, or GitHub Actions).

‍

Trivy Scanner Different Applications Diagram

Find out more about the different Operating Systems and Application Dependencies that Trivy can scan here:

https://aquasecurity.github.io/trivy/v0.28.1/docs/vulnerability/detection/os/

https://aquasecurity.github.io/trivy/v0.28.1/docs/vulnerability/detection/language/

‍

Is Trivy Free?

Yes, Trivy is 100% free since it is an open-source project. Aqua, the team behind Trivy, is committed to ensuring that this project remains open-source since it guarantees the maintenance of high-quality code and participation in other open-source projects.

‍

How to Integrate Trivy into an Existing GitLab CI Pipeline

There are two approaches to integrating the Trivy scanner into GitLab CI. Firstly, GitLab's CI offers a security scanner integration based on Trivy if you host your containers with GitLab’s Container Registry. This is probably the easiest solution to integrate Trivy into a CI Pipeline. You can find more information with this link: https://docs.gitlab.com/ee/user/application_security/container_scanning/.

On the other hand, If you are using another container registry, in our case Google Container Registry, things will work a bit differently but won't be a big challenge to accomplish.

Please note that for this blog post, we cover integrating Trivy into an existing GitLab CI pipeline using Google Container Registry. However, we recommend using Google Artifact Registry, which is the current offering for storing, managing, and securing your build artifacts on Google Cloud.

The link to the repository used:

https://gitlab.com/bluelightco/blog-examples/trivy

We have a Dockerfile that builds an Apache server based on the container httpd and files in public-html.

We ran the building stage and performed a quick vulnerability scan with Trivy Standalone before pushing the image to our container registry.

‍

GitLab CI with Trivy connection full diagram

Setting Up a Service Account on GCP

For Trivy to scan from a private container registry such as GCR, you must create a service account with read permissions on the container registry.

Below are the instructions you need to create the required permissions:

https://cloud.google.com/container-registry/docs/access-control

Once you create the service account, you need to create a key in .json:

https://cloud.google.com/container-registry/docs/advanced-authentication#json-key

We used the same key to pull/build, push and run the scanning job from Trivy.

Export that key so you can use it as a variable on GitLab CI:

  xclip -selection clipboard < /home/user/Downloads/my_key.json

From the repository, navigate to the Settings > CI/CD > Variables:

Create a new variable and paste your key from the xclip command.

Main Screen of the Variable Settings Menu where you can add a new variable

‍

Modifying .gitlab-ci.yml to Integrate the Scan Job by Trivy

Right before we push our image, this is where we will be running Trivy to perform a quick scan for potential vulnerability. If there's a vulnerability with a severity marked as Critical, we want the pipeline to fail so we can fix this issue:

  - trivy image --no-progress --exit-code 0 --severity HIGH us.gcr.io/$GOOGLE_PROJECT_ID/my-apache2:$CI_COMMIT_SHORT_SHA
- trivy image --no-progress --exit-code 1 --severity CRITICAL us.gcr.io/$GOOGLE_PROJECT_ID/my-apache2:$CI_COMMIT_SHORT_SHA

Flag options:

--no-progress suppresses the progress bar generated to keep the terminal quiet;
--ignore-unfixed by default, Trivy detects unpatched/unfixed vulnerabilities. This means you can't fix these vulnerabilities even if you update all packages;
--severity allows you to set more filters and only show High or Critical vulnerabilities;
--exit-code when set to 1, this will allow the entire pipeline to fail if it finds a critical vulnerability;
.trivyignore not used our case, but if you believe that a vulnerability should be ignored.

Source:

.gitlab-ci.yml

‍

Viewing Reports from Trivy

Your pipeline is now running. Trivy can now scan the container image that you are about to push to GCR.

The pipeline has failed, let's review the GitLab Runner:

  $ trivy image --no-progress --light --exit-code 1 --severity CRITICAL us.gcr.io/$GOOGLE_PROJECT_ID/my-apache2:$CI_COMMIT_SHORT_SHA
2021-05-17T15:57:16.094Z	INFO	Detecting Debian vulnerabilities...
2021-05-17T15:57:16.109Z	INFO	Trivy skips scanning programming language libraries because no supported file was detected
us.gcr.io/florian-test-297317/my-apache2:b971d877 (debian 10.9)
===============================================================
Total: 2 (CRITICAL: 2)
+-------------+------------------+----------+-------------------+---------------+
|   LIBRARY   | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |
+-------------+------------------+----------+-------------------+---------------+
| libgnutls30 | CVE-2021-20231   | CRITICAL | 3.6.7-4+deb10u6   |               |
+             +------------------+          +                   +---------------+
|             | CVE-2021-20232   |          |                   |               |
+-------------+------------------+----------+-------------------+---------------+
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1

This is the expected behavior with the flag --exit code 1 that we set if it finds a Critical vulnerability.

‍

Bonus Tip - Scan images of a Kubernetes Resource with Trivy

Trivy allows you to extend the CLI without changing the Trivy codebase thanks to plugins. Inspired by kubectl and Helm, you can find more info on installing and using plugins here.

A plugin that is great to use is trivy-plugin-kubectl, which allows you to scan a pod/job or deployment on your Kubernetes.

Example:

  trivy kubectl pod grafana-65d486b746-dsk2r

2021-05-17T16:02:39.222-0700	INFO	Detecting Alpine vulnerabilities...
2021-05-17T16:02:39.225-0700	INFO	Detecting gobinary vulnerabilities...

grafana/grafana:7.5.3 (alpine 3.12.6)
=====================================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+-----------+------------------+----------+-------------------+---------------+---------------------------------------+
|  LIBRARY  | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+-----------+------------------+----------+-------------------+---------------+---------------------------------------+
| apk-tools | CVE-2021-30139   | HIGH     | 2.10.5-r1         | 2.10.6-r0     | In Alpine Linux apk-tools             |
|           |                  |          |                   |               | before 2.12.5, the tarball            |
|           |                  |          |                   |               | parser allows a buffer...             |
|           |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-30139 |
+-----------+------------------+----------+-------------------+---------------+---------------------------------------+

usr/share/grafana/bin/grafana-cli
=================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)


usr/share/grafana/bin/grafana-server
====================================
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 1, CRITICAL: 0)

+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
|             LIBRARY              | VULNERABILITY ID | SEVERITY |          INSTALLED VERSION           | FIXED VERSION |                 TITLE                 |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
| github.com/apache/thrift         | CVE-2020-13949   | HIGH     | v0.13.0                              | v0.14.0       | libthrift: potential DoS when         |
|                                  |                  |          |                                      |               | processing untrusted payloads         |
|                                  |                  |          |                                      |               | -->avd.aquasec.com/nvd/cve-2020-13949 |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
| github.com/prometheus/prometheus | CVE-2019-3826    | MEDIUM   | v1.8.2-0.20201105135750-00f16d1ac3a4 | v2.7.1        | prometheus: Stored DOM                |
|                                  |                  |          |                                      |               | cross-site scripting (XSS)            |
|                                  |                  |          |                                      |               | attack via crafted URL                |
|                                  |                  |          |                                      |               | -->avd.aquasec.com/nvd/cve-2019-3826  |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
2021-05-17T16:02:40.639-0700	WARN	OS is not detected and vulnerabilities in OS packages are not detected.
2021-05-17T16:02:40.639-0700	INFO	Trivy skips scanning programming language libraries because no supported file was detected
2021-05-17T16:02:41.933-0700	INFO	Detecting Alpine vulnerabilities...
2021-05-17T16:02:41.933-0700	INFO	Detecting gobinary vulnerabilities...

docker.io/grafana/grafana:7.5.3 (alpine 3.12.6)
===============================================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+-----------+------------------+----------+-------------------+---------------+---------------------------------------+
|  LIBRARY  | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |
+-----------+------------------+----------+-------------------+---------------+---------------------------------------+
| apk-tools | CVE-2021-30139   | HIGH     | 2.10.5-r1         | 2.10.6-r0     | In Alpine Linux apk-tools             |
|           |                  |          |                   |               | before 2.12.5, the tarball            |
|           |                  |          |                   |               | parser allows a buffer...             |
|           |                  |          |                   |               | -->avd.aquasec.com/nvd/cve-2021-30139 |
+-----------+------------------+----------+-------------------+---------------+---------------------------------------+

usr/share/grafana/bin/grafana-cli
=================================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)


usr/share/grafana/bin/grafana-server
====================================
Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 1, CRITICAL: 0)

+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
|             LIBRARY              | VULNERABILITY ID | SEVERITY |          INSTALLED VERSION           | FIXED VERSION |                 TITLE                 |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
| github.com/apache/thrift         | CVE-2020-13949   | HIGH     | v0.13.0                              | v0.14.0       | libthrift: potential DoS when         |
|                                  |                  |          |                                      |               | processing untrusted payloads         |
|                                  |                  |          |                                      |               | -->avd.aquasec.com/nvd/cve-2020-13949 |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
| github.com/prometheus/prometheus | CVE-2019-3826    | MEDIUM   | v1.8.2-0.20201105135750-00f16d1ac3a4 | v2.7.1        | prometheus: Stored DOM                |
|                                  |                  |          |                                      |               | cross-site scripting (XSS)            |
|                                  |                  |          |                                      |               | attack via crafted URL                |
|                                  |                  |          |                                      |               | -->avd.aquasec.com/nvd/cve-2019-3826  |
+----------------------------------+------------------+----------+--------------------------------------+---------------+---------------------------------------+
2021-05-17T16:02:43.327-0700	WARN	OS is not detected and vulnerabilities in OS packages are not detected.
2021-05-17T16:02:43.327-0700	INFO	Trivy skips scanning programming language libraries because no supported file was detected

‍

Conclusion

Running a security scanner for container images isn't something you only want to do when you're about to merge your changes to Production. The best practice is to run it on a featured branch, so you have time to fix any potential security issues you might have with your build. It is part of the DevSecOps methodology covered in our blog post, and we believe it will be resourceful for DevOps and security teams.

‍