How to Install Grafana Loki Stack using AWS S3 Bucket

For any production environment, having a proper logging and monitoring system is vital. This is down to the fact that downtime can lead to unwanted bottlenecks in such an environment and even loss of business reputation and revenue. As a result, most organizations that utilize DevOps principles and tools require a logging system to cover the shortcomings of Prometheus, which is an efficient production system monitoring tool but lacks logging capabilities.

Grafana Loki is one of the tools that help companies maintain a centralized logging system. The importance of having a centralized logging system in a production environment is that it allows you to address issues such as downtime and identify potential issues that need resolution. Loki started at Grafana Labs in 2018 and was announced at the KubeCon in Seattle in December 2018. In June 2019, the first beta version 0.1.0 was released.

‍

What is Grafana Loki?

Is Grafana Loki free?

Advantages of using Grafana Loki

How to Install Grafana Loki stack

Launching Grafana Loki stack

How to access the Grafana Web UI

Handling queries with Grafana Loki

Conclusion

‍

What is Grafana Loki?

Loki is a multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. Log data is compressed and stored in chunks in object stores such as AWS S3 and GCS or can be even saved locally on a file system.

‍

Is Grafana Loki Free?

Loki is an open-source project aimed at helping DevOps professionals install and run Grafana without much hassle. While Grafana has pricing plan for hosting/storing your log data, Loki isn’t affected by this pricing plan. As such, it remains a free to use tool.

‍

Advantages of using Grafana Loki:

Some key benefits of using Grafana Loki versus competitors such as Graylog and Datadog, to name a few, are:

‍Lightweight: By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.

‍Object storage: Loki can be set up with Amazon S3 or GCS, which tends to be cheaper than block storage. The link below will show you the different ways to store the index and chunks:
‍https://grafana.com/docs/loki/latest/operations/storage/

‍Multi-tenancy support: It's supported through a tenant ID so that the tenant's data is stored separately.
‍https://grafana.com/docs/loki/latest/overview/#multi-tenancy

‍Native Grafana support: from Grafana 6.0, you can review metrics and logs all in one place.

‍

How to Install Grafana Loki Stack

Our focus in this article is Loki Stack which consists of 3 main components:

Grafana for querying and displaying the logs.
Loki is the main server responsible for storing the logs and processing queries.
Promtail is the agent responsible for gathering logs and pushing them to Loki.

We will be using minikube and Flux version 2 to deploy Loki Stack with a Helm chart. By default, Loki will store the chunks on the file system that shares the pod's lifetime. We will change that and use an AWS S3 Bucket to store this kind of data.

You can use Fluentbit instead of Promtail if you want to, but we have noticed that Promtail is well set out of the box and can discover all labels on your Kubernetes Cluster.

Access to the repo used for this post can be found with the link below:

https://github.com/bluelightco/blog-loki

‍

Launching Grafana Loki Stack

Deploy with a helm chart.

More information about the helm chart used can be found with the link below:

https://github.com/grafana/helm-charts/tree/main/charts/loki-stack

https://github.com/grafana/helm-charts/blob/main/charts/loki-stack/values.yaml

Below is the Helm values.yaml file that we will be using:

  values:
  loki:
    env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: iam-loki-s3
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: iam-loki-s3
            key: AWS_SECRET_ACCESS_KEY
    config:
      schema_config:
        configs:
          - from: 2021-05-12
            store: boltdb-shipper
            object_store: s3
            schema: v11
            index:
              prefix: loki_index_
              period: 24h
      storage_config:
        aws:
          s3: s3://us-west-2/bluelightco-loki
          s3forcepathstyle: true
          bucketnames: bluelightco-loki
          region: us-west-2
          insecure: false
          sse_encryption: false
        boltdb_shipper:
          shared_store: s3
          cache_ttl: 24h
  grafana:
    enabled: true
    sidecar:
      datasources:
        enabled: true
    image:
      tag: latest
    grafana.ini:
      users:
        default_theme: light

Since we will be using an AWS S3 Bucket to store our compressed log data in chunks, we created an IAM user with write and read permission to our S3 Bucket. A secret will need to be created so Loki can use those credentials mentioned in the Helm values above.

Run the following command with your own IAM credentials:

  kubectl create secret generic iam-loki-s3 --from-literal=AWS_ACCESS_KEY_ID='ABC123456' --from-literal=AWS_SECRET_ACCESS_KEY='ABC123456' -n default

‍

How to Access the Grafana Web UI

To access Grafana web UI, you will have to run the following command:

  kubectl port-forward svc/loki-grafana -n default 8080:80

Browse the following URL: http://localhost:8080/

To retrieve the password, run:

  kubectl get secret --namespace default loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

‍

Handling Queries with Grafana Loki

Now that you are logged in to Grafana, Loki has been automatically added as a data source thanks to the helm chart Loki-Stack, making it easy to use Loki right away.

On the Grafana web UI, select "Explore" from the side panel.

Grafana Web User Interface "Explore" option on the main menu

‍

You will be welcomed with the following Log Browser. Instead of manually writing your own query, you can use this menu to help you to build it.

Log Browser on Grafana's Explore Section

‍

Let's take a look at the logs for the redis-master-0 pod, click on pod, and select redis-master-0:

‍

If you are hoping to run some advanced queries, we highly suggest looking at the Log Query Language (LogQL) documentation provided by Grafana here.

‍

Conclusion

Most businesses want a simple and cost-effective solution, and Loki provides all of that into a solution that collects, stores, and analyzes log files from apps and services. It gives you a new approach to central log management. The simplicity of setting it up and integrating with Grafana allows you to monitor metrics and logs all in one place, making it a solid choice.

How to Install Grafana Loki Stack using AWS S3 Bucket

Blog

Table of Contents