How to Install Grafana Loki Stack using AWS S3 Bucket

Written By

Chase Bolt

Blog

For any production environment, having a proper logging and monitoring system is vital. This is down to the fact that downtime can lead to unwanted bottlenecks in such an environment and even loss of business reputation and revenue. As a result, most organizations that utilize DevOps principles and tools require a logging system to cover the shortcomings of Prometheus, which is an efficient production system monitoring tool but lacks logging capabilities. 

Grafana Loki is one of the tools that help companies maintain a centralized logging system. The importance of having a centralized logging system in a production environment is that it allows you to address issues such as downtime and identify potential issues that need resolution. Loki started at Grafana Labs in 2018 and was announced at the KubeCon in Seattle in December 2018. In June 2019, the first beta version 0.1.0 was released.

Table of Contents

               Launching Grafana Loki stack

               How to access the Grafana Web UI

               Handling queries with Grafana Loki

What is Grafana Loki?

Loki is a multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. Log data is compressed and stored in chunks in object stores such as AWS S3 and GCS or can be even saved locally on a file system.

Is Grafana Loki Free?

Loki is an open-source project aimed at helping DevOps professionals install and run Grafana without much hassle. While Grafana has pricing plan for hosting/storing your log data, Loki isn’t affected by this pricing plan. As such, it remains a free to use tool.

Advantages of using Grafana Loki:

Some key benefits of using Grafana Loki versus competitors such as Graylog and Datadog, to name a few, are:

  • Lightweight: By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
  • Native Grafana support: from Grafana 6.0, you can review metrics and logs all in one place.

How to Install Grafana Loki Stack

Our focus in this article is Loki Stack which consists of 3 main components:

  • Grafana for querying and displaying the logs.
  • Loki is the main server responsible for storing the logs and processing queries.
  • Promtail is the agent responsible for gathering logs and pushing them to Loki.
Loki Stack Main Components Diagram

We will be using minikube and Flux version 2 to deploy Loki Stack with a Helm chart. By default, Loki will store the chunks on the file system that shares the pod's lifetime. We will change that and use an AWS S3 Bucket to store this kind of data.

You can use Fluentbit instead of Promtail if you want to, but we have noticed that Promtail is well set out of the box and can discover all labels on your Kubernetes Cluster.

Access to the repo used for this post can be found with the link below:

https://github.com/bluelightco/blog-loki

Launching Grafana Loki Stack

Deploy with a helm chart.

More information about the helm chart used can be found with the link below:

https://github.com/grafana/helm-charts/tree/main/charts/loki-stack

https://github.com/grafana/helm-charts/blob/main/charts/loki-stack/values.yaml

Below is the Helm values.yaml  file that we will be using:

  values:
  loki:
    env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: iam-loki-s3
            key: AWS_ACCESS_KEY_ID
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: iam-loki-s3
            key: AWS_SECRET_ACCESS_KEY
    config:
      schema_config:
        configs:
          - from: 2021-05-12
            store: boltdb-shipper
            object_store: s3
            schema: v11
            index:
              prefix: loki_index_
              period: 24h
      storage_config:
        aws:
          s3: s3://us-west-2/bluelightco-loki
          s3forcepathstyle: true
          bucketnames: bluelightco-loki
          region: us-west-2
          insecure: false
          sse_encryption: false
        boltdb_shipper:
          shared_store: s3
          cache_ttl: 24h
  grafana:
    enabled: true
    sidecar:
      datasources:
        enabled: true
    image:
      tag: latest
    grafana.ini:
      users:
        default_theme: light

Since we will be using an AWS S3 Bucket to store our compressed log data in chunks, we created an IAM user with write and read permission to our S3 Bucket. A secret will need to be created so Loki can use those credentials mentioned in the Helm values above.

Run the following command with your own IAM credentials:

  kubectl create secret generic iam-loki-s3 --from-literal=AWS_ACCESS_KEY_ID='ABC123456' --from-literal=AWS_SECRET_ACCESS_KEY='ABC123456' -n default

How to Access the Grafana Web UI

To access Grafana web UI, you will have to run the following command:

  kubectl port-forward svc/loki-grafana -n default 8080:80

Browse the following URL: http://localhost:8080/

Log in with the username admin

To retrieve the password, run:

  kubectl get secret --namespace default loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Handling Queries with Grafana Loki

Now that you are logged in to Grafana, Loki has been automatically added as a data source thanks to the helm chart Loki-Stack, making it easy to use Loki right away.

On the Grafana web UI, select "Explore" from the side panel.

Grafana Web User Interface "Explore" option on the main menu

You will be welcomed with the following Log Browser. Instead of manually writing your own query, you can use this menu to help you to build it.

Log Browser on Grafana's Explore Section

Let's take a look at the logs for the redis-master-0 pod, click on pod, and select redis-master-0:

Logs for redis-master- main dashboard

Logs for redis-master- main list

If you are hoping to run some advanced queries, we highly suggest looking at the Log Query Language (LogQL) documentation provided by Grafana here.

Conclusion

Most businesses want a simple and cost-effective solution, and Loki provides all of that into a solution that collects, stores, and analyzes log files from apps and services. It gives you a new approach to central log management. The simplicity of setting it up and integrating with Grafana allows you to monitor metrics and logs all in one place, making it a solid choice.

You may also be interested in:

The Complete Python Developer Salary Guide for 2022

Diagrams as Code: The Complete How-to-Use Guide

Nearshore Staff Augmentation: Top 4 Benefits for Businesses

What is Kubecost: The Complete Guide

The Complete React Developer Salary Guide for 2022

A Detailed Overview of the Top 11 AWS Certification Courses

The Ultimate DevOps Hourly Rate Guide for 2022

Redux Toolkit with Typescript: How to Get Started

Bluelight Consulting is a nearshore DevOps & Software Outsourcing company that helps startups, SaaS, and enterprises with cutting-edge solutions.

More cost-effective than hiring in-house, with Nearshore Boost, our nearshore software development service, you can ensure your business stays competitive with an expanded team and a bigger global presence, you can be flexible as you respond to your customers’ needs.

Learn more about our services by booking a free consultation with us today!

Let us solve your business’ biggest challenges

Book a free Consultation
Save 50+ hours of project time per developer on interviewing.
Tell us the skills you need and we'll find the best developer for your needs in days, not weeks.

Discuss your project with us today!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.