Deploying Grafana Loki on Kubernetes

I deployed Loki as part of an observability platform on a seven-node, AlmaLinux bare-metal Kubernetes cluster, using Rook Ceph for object storage. The goal was to centralize logs from infrastructure services, including Ceph and HAProxy, and query them through Grafana.

Technical revision — October 2026: the original article incorrectly described Mimir as Loki’s log-storage backend and reversed the release/chart arguments in the installation command. The architecture and commands below correct those errors. The versioned example is a historical reference, not a recommendation to start a new production deployment on these releases.

Where logs and metrics go

Loki ingests and queries logs. Mimir ingests and queries Prometheus metrics. Each uses its own buckets in the S3-compatible Ceph object store; Grafana queries both as separate data sources. Mimir is not in Loki’s log-storage path.

1
2
3

Log files → collector → authenticating gateway → Loki → Ceph S3 log buckets
Prometheus → authenticating gateway → Mimir → Ceph S3 metric buckets
Grafana → Loki (LogQL) and Mimir (PromQL)

Loki’s distributors accept writes and route them to ingesters. Ingesters maintain recent data and write chunks to object storage. Queriers retrieve log data from ingesters and storage; query frontends and schedulers coordinate query work. The compactor handles index compaction and, when explicitly configured, retention.

Labels identify streams. High-cardinality labels can greatly increase stream count and index overhead; they should not be described as creating one index shard per label. Keep labels bounded and use log content or structured metadata for values such as request IDs.

Versioned configuration

The downloadable values.yaml targets grafana/loki chart 6.24.0, Loki 3.3.2, with a Kubernetes 1.30 rendering target. These are reference versions selected for this revision; the original deployment’s exact chart version was not recorded in the article. The example was checked with Helm 3.17.3 using lint and template rendering. It has not been installed into a cluster during this revision.

For a new deployment, select supported versions and follow the current Loki installation guide. The chart has since moved to the Grafana Community repository; review its migration instructions instead of applying these historical values to a current chart.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102

# Historical reference: grafana/loki chart 6.24.0 (Loki 3.3.2).
# Rendered locally; adapt endpoint, Secret, PVCs and authentication before use.
deploymentMode: Distributed
global:
  clusterDomain: cluster.local
  dnsService: kube-dns
  dnsNamespace: kube-system
loki:
  auth_enabled: true
  commonConfig:
    replication_factor: 3
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  storage:
    type: s3
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    s3:
      endpoint: https://s3.example.com
      region: us-east-1
      accessKeyId: ${AWS_ACCESS_KEY_ID}
      secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
      s3ForcePathStyle: true
      insecure: false
ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: false
  persistence:
    enabled: true
    claims:
      - name: data
        size: 10Gi
        storageClass: rook-ceph-block
  extraArgs: &expandEnv
    - -config.expand-env=true
  extraEnvFrom: &storageEnv
    - secretRef:
        name: loki-s3
querier:
  replicas: 2
  maxUnavailable: 1
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
queryFrontend:
  replicas: 2
  maxUnavailable: 1
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
queryScheduler:
  replicas: 2
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
distributor:
  replicas: 2
  maxUnavailable: 1
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
compactor:
  replicas: 1
  persistence:
    enabled: true
    claims:
      - name: data
        size: 10Gi
        storageClass: rook-ceph-block
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
indexGateway:
  replicas: 2
  maxUnavailable: 1
  persistence:
    enabled: true
    size: 10Gi
    storageClass: rook-ceph-block
  extraArgs: *expandEnv
  extraEnvFrom: *storageEnv
gateway:
  replicas: 2
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0
singleBinary:
  replicas: 0
minio:
  enabled: false
lokiCanary:
  enabled: false
test:
  enabled: false

Important choices in this example:

• Three ingesters and replication factor three require enough schedulable nodes. The chart’s anti-affinity and your storage topology must be checked; replicas alone do not prove fault tolerance.
• Zone-aware replication is explicitly disabled. This is a host-level reference, not a claim of resilience across availability zones.
• The compactor runs as a singleton. Its responsibilities do not justify one replica per cluster node. Retention is not enabled by this example; define and test a retention policy before relying on automatic deletion. See Loki retention.
• PVCs reference rook-ceph-block. Confirm that StorageClass, free space, placement and reclaim policy before use.
• Replica counts, cache defaults, resources and disruption budgets require load testing. They are not a sizing recommendation for a 300 TB storage cluster.
• Canary and Helm tests are disabled until an authenticated collection/query path is configured. Enable equivalent end-to-end monitoring before production use.

Object storage and credentials

Provision a dedicated Ceph user with access limited to Loki’s buckets. With a Rook CephObjectStoreUser named loki in object store object-store, inspect the generated Secret’s metadata with:

1

kubectl -n rook-ceph describe secret rook-ceph-object-user-object-store-loki

That Secret belongs to the rook-ceph namespace. A Secret reference cannot cross namespaces: use your secret-management workflow to supply a separate loki-s3 Secret in namespace loki, containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The chart injects these variables and expands them in the Loki configuration. Do not commit key values or rendered configuration containing real credentials.

Configure a trusted HTTPS endpoint in the values file. For an internal CA, install or mount the required CA and configure the client trust settings for the chosen version. insecure: false assumes a certificate chain the client trusts; it does not install your CA.

Bucket commands must point explicitly to Ceph, not the default AWS S3 endpoint. The following Bash example assumes an AWS CLI profile ceph-loki that already contains the restricted Ceph user’s credentials:

1
2
3
4
5
6
7

export AWS_PROFILE=ceph-loki
export CEPH_S3_ENDPOINT=https://s3.example.com
# If the endpoint uses an internal CA:
# export AWS_CA_BUNDLE=/path/to/internal-ca.pem
for bucket in loki-chunks loki-ruler loki-admin; do
  aws --endpoint-url "$CEPH_S3_ENDPOINT" --region us-east-1 s3 mb "s3://$bucket"
done

Use the same endpoint, bucket names and region/signature settings in Loki. Create buckets once, and verify existing buckets rather than assuming a failed creation command means the storage is unusable. TLS verification remains enabled.

Render and install deliberately

In an empty working directory, download values.yaml, adapt its endpoint and DNS settings, and review the manifests:

1
2
3

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm template loki grafana/loki --version 6.24.0   --namespace loki --kube-version 1.30.0 -f values.yaml > rendered.yaml

The correct command order is release name, chart name, followed by flags. After verifying the target cluster, namespace, Secrets, storage, resources and authentication design, the corresponding historical installation command is:

1

helm upgrade --install loki grafana/loki --version 6.24.0   --namespace loki --create-namespace -f values.yaml --wait --timeout 10m

helm template verifies chart rendering; it does not validate storage connectivity, application configuration at runtime or the cluster’s capacity to schedule the pods.

Authentication, collection and verification

auth_enabled: true enables Loki’s multi-tenant mode. Clients must supply the appropriate X-Scope-OrgID, but that header alone is not proof of identity. Put an authenticating and authorizing proxy in front of the API, and have it set or validate tenant identity. The chart’s routing gateway does not automatically establish this security boundary. Keep direct service access restricted. See Loki authentication.

The original collection example used Promtail. It reached end of life on March 2, 2026. Use a supported collector such as Grafana Alloy for a new setup, following the Promtail migration guidance. Give the collector only the filesystem permissions it needs; running as root is not a universal prerequisite.

Add Loki’s authenticated query endpoint as a Grafana data source. Validate ingestion and querying for a known log line, tenant separation, pod placement, PVC binding and object-store access. Then test restarts and node failure under representative load. Those checks are necessary before claiming high availability or choosing the final replica counts.