So I actually forgot about this post; I thought I had taken it out of draft status but oopsie-daisy. In the end, I only got around to adding a Grafana stack including Loki to the DOKS cluster.

In my defence, it has been very busy. I’ll be posting about that soon.

Here are some of the extra bibs and bobs that I put on my DOKS 2021 Challenge cluster.

Some extra tools are needed here, or at least, I make use of them.

  • kubectl, ‘lets you control Kubernetes clusters’
  • doctl, optional ‘the official CLI tool for interacting with DO resources’
  • k9s, optional a helpful TUI for interacting with k8s resources

You can download the kubeconfig for the cluster from the DO gui, doctl or extract it from Terraform state. I’ll be using doctl just because I wanted to try it.

$ brew install doctl
==> Downloading https://ghcr.io/v2/homebrew/core/doctl/manifests/1.67.0
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/doctl/blobs/sha256:7c235dbbf49483325e38b76cb72263ec9301e1a6ffd958ee5d15
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:7c235dbbf49483325e38b76cb72263ec930
######################################################################## 100.0%
==> Pouring doctl--1.67.0.x86_64_linux.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
🍺  /home/linuxbrew/.linuxbrew/Cellar/doctl/1.67.0: 9 files, 24.6MB

$ doctl auth init
Please authenticate doctl for use with your DigitalOcean account. You can generate a token in the control panel at https://cloud.digitalocean.com/account/api/tokens

Enter your access token:
Validating token... OK

$ doctl kubernetes cluster kubeconfig save 898db296-575e-4112-a651-4d0c429c5b28
Notice: Adding cluster credentials to kubeconfig file found in "/home/liam/.kube/config"
Notice: Setting current-context to do-sgp1-do-k8s-2021

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Okay, good to go.

Loki / Grafana stack

Update main.tf and apply.

resource "helm_release" "loki" {
  name       = "loki"
  repository = "https://grafana.github.io/helm-charts"
  chart      = "loki-stack"
  version    = "2.5.0"

  namespace        = "loki"
  create_namespace = true

  set {
    name  = "grafana.enabled"
    value = "true"
  }

  set {
    name  = "prometheus.enabled"
    value = "true"
  }
}

Get the admin password for the initial Grafana login, and port forward so that you have access to Grafana locally.

$ kubectl get secret --namespace loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
<password output here>

$ kubectl port-forward --namespace loki service/loki-grafana 8080:80
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000

Hit http://localhost:8080 and you should be staring at a Grafana login page. Success! Deep diving on Grafana is a bit much for this post, but have a play around in the explore tab on the left after you login. You’ll already have logs (thanks to Loki) and cluster metrics (thanks to Prometheus) available to view.

One fun thing I did was check what output there was in Falco when I opened up a shell on my Loki pod. I used the LogQL search term {job="falco/falco"}, i.e. give me all the logs for the falco job in the falco namespace, hit search and voila!

2021-11-27T07:38:52.832748533Z stdout F 07:38:52.830148939: Notice A shell was spawned in a container with an attached terminal (user=<NA> user_loginuid=-1 k8s.ns=loki k8s.pod=loki-0 container=269597121f05 shell=sh parent=<NA> cmdline=sh terminal=34816 container_id=269597121f05 image=docker.io/grafana/loki) k8s.ns=loki k8s.pod=loki-0 container=269597121f05
2021-11-27T07:38:52.832551682Z stdout F 07:38:52.830014807: Notice A shell was spawned in a container with an attached terminal (user=<NA> user_loginuid=-1 k8s.ns=loki k8s.pod=loki-0 container=269597121f05 shell=sh parent=<NA> cmdline=sh -c command -v bash >/dev/null && exec bash || exec sh terminal=34816 container_id=269597121f05 image=docker.io/grafana/loki) k8s.ns=loki k8s.pod=loki-0 container=269597121f05

Maybe something like count_over_time({job="falco/falco"} |= "A shell was spawned in a container with an attached terminal"[1m]) will be a useful one to keep an eye on. 🤔