This document details the deployment of Mimir, a horizontally scalable, highly available time series database, on a seven-node on-premises Kubernetes cluster. The cluster, running AlmaLinux on bare-metal servers with a combined storage capacity of approximately 300TB, leverages Rook Ceph for storage management. This deployment will serve as a central repository for metrics gathered from multiple Prometheus servers, providing a robust and scalable solution for long-term metric storage and analysis. The configuration emphasizes high availability and resilience, leveraging the resources of the seven-node cluster to ensure optimal performance and minimize potential single points of failure. Specific configurations and considerations regarding resource allocation, storage management, and security are outlined in the following sections.
1. Helm Repository Addition and Update:
1
2
| helm repo add grafana https://grafana.github.io/helm-charts # Adds the Grafana Helm chart repository.
helm repo update # Updates the local Helm repository cache.
|
2. Custom values.yaml Configuration:
This values.yaml file customizes the Mimir deployment. Key customizations are noted below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
| metaMonitoring:
serviceMonitor:
enabled: true # Enables ServiceMonitor for monitoring Mimir itself.
grafanaAgent:
enabled: true # Enables Grafana Agent for metrics collection.
installOperator: false # Does not install the Grafana Agent operator.
logs:
remote:
url: "http://loki-gateway.loki.svc/loki/api/v1/push" # Sends logs to a Loki instance.
metrics:
remote:
url: "https://kube-prometheus-stack-prometheus.monitoring.svc:9090/api/v1/push" # Sends metrics to a Prometheus instance.
headers:
X-Scope-OrgID: metamonitoring # Adds a custom header for organization ID.
global:
dnsService: coredns # Specifies CoreDNS as the DNS service.
dnsNamespace: kube-system # Specifies the namespace for CoreDNS.
clusterDomain: sp-logs. # Sets the cluster domain.
mimir:
structuredConfig:
limits:
max_global_series_per_user: 1000000 # Significantly increases the limit on the number of time series per user. This is a crucial customization to avoid throttling issues.
common:
storage:
backend: s3 # Configures S3 as the storage backend.
s3:
endpoint: rook-ceph-rgw-object-store.rook-ceph.svc.sp-logs # Uses Rook Ceph as the S3 compatible storage.
region: us-east-1 # Required even though we are using Ceph
secret_access_key: <secret_key> # **IMPORTANT:** Replace with your actual secret key. This is a security risk if committed to version control. Use secrets management.
access_key_id: <access_key> # **IMPORTANT:** Replace with your actual access key. Same security risk as above.
insecure: true # **WARNING:** Disables TLS verification for the S3 connection. This is a significant security risk and should only be used in controlled environments.
blocks_storage:
s3:
bucket_name: mimir-blocks
alertmanager_storage:
s3:
bucket_name: mimir-alertmanager
ruler_storage:
s3:
bucket_name: mimir-ruler
# The following sections define resource limits and replicas for different Mimir components. The replica counts (especially `ingester` and `index-cache` set to 7) are customized to match the 7-node cluster. Adjust based on your needs. Also note the generous resource limits, especially for `ingester`. These should be adjusted based on the expected load.
alertmanager:
persistentVolume:
enabled: true
replicas: 2
resources:
limits:
memory: 1.4Gi
requests:
cpu: 1
memory: 1Gi
statefulSet:
enabled: true
compactor:
persistentVolume:
size: 20Gi
resources:
limits:
memory: 2.1Gi
requests:
cpu: 1
memory: 1.5Gi
distributor:
replicas: 2
resources:
limits:
memory: 5.7Gi
requests:
cpu: 2
memory: 4Gi
ingester:
persistentVolume:
size: 50Gi
replicas: 7 # 7 replicas, matching the number of nodes in the cluster.
resources:
limits:
memory: 12Gi # High memory limit for ingester.
requests:
cpu: 3.5
memory: 8Gi
topologySpreadConstraints: {}
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: target # support for enterprise.legacyLabels
operator: In
values:
- ingester
topologyKey: 'kubernetes.io/hostname'
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- ingester
topologyKey: 'kubernetes.io/hostname'
zoneAwareReplication:
topologyKey: 'kubernetes.io/hostname'
# other Mimir configurations:
admin-cache:
enabled: true
replicas: 2
chunks-cache:
enabled: true
replicas: 2
index-cache:
enabled: true
replicas: 7 # maximized to use all available nodes
metadata-cache:
enabled: true
results-cache:
enabled: true
replicas: 2
minio:
enabled: false
overrides_exporter:
replicas: 1
resources:
limits:
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
querier:
replicas: 1
resources:
limits:
memory: 5.6Gi
requests:
cpu: 2
memory: 4Gi
query_frontend:
replicas: 1
resources:
limits:
memory: 2.8Gi
requests:
cpu: 2
memory: 2Gi
ruler:
replicas: 1
resources:
limits:
memory: 2.8Gi
requests:
cpu: 1
memory: 2Gi
store_gateway:
persistentVolume:
size: 10Gi
replicas: 2
resources:
limits:
memory: 2.1Gi
requests:
cpu: 1
memory: 1.5Gi
topologySpreadConstraints: {}
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: target # support for enterprise.legacyLabels
operator: In
values:
- store-gateway
topologyKey: 'kubernetes.io/hostname'
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- store-gateway
topologyKey: 'kubernetes.io/hostname'
zoneAwareReplication:
topologyKey: 'kubernetes.io/hostname'
nginx:
replicas: 1
resources:
limits:
memory: 731Mi
requests:
cpu: 1
memory: 512Mi
ingress:
enabled: true
annotations: {}
hosts:
- host: mimir.mydomain
paths:
- path: /
pathType: ImplementationSpecific
tls:
- hosts:
- mimir.mydomain
|
3. Ceph Object Store User Creation:
This section describes creating a user for Mimir in the Rook Ceph object store. Using a Kubernetes manifest is the preferred approach over radosgw-admin.
1
2
3
4
5
6
7
8
9
10
11
| apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
name: mimir
namespace: rook-ceph
spec:
store: object-store
displayName: mimir
capabilities:
user: "*"
bucket: "*"
|
1
| k apply -f CephObjectStoreUser.yaml
|
4. Retrieving S3 Access Keys:
The command below retrieves the S3 access keys from the Kubernetes secret.
1
| k -n rook-ceph get secret rook-ceph-object-user-object-store-loki -oyaml
|
5. Bucket Creation:
The following commands create the necessary S3 buckets. Note the use of --no-verify-ssl.
1
2
3
| aws --no-verify-ssl s3 mb s3://mimir-blocks --region us-east-1
aws --no-verify-ssl s3 mb s3://mimir-ruler --region us-east-1
aws --no-verify-ssl s3 mb s3://mimir-alertmanager --region us-east-1
|
6. Mimir Installation:
This command installs Mimir using the customized values.yaml file.
1
| helm install --namespace mimir grafana/mimir mimir -f values.yaml
|
Security Considerations:
- Hardcoded Credentials: The
values.yaml file contains hardcoded AWS access keys. This is a major security vulnerability. Use Kubernetes Secrets to manage these credentials securely. insecure: true: Disabling TLS verification for the S3 connection is highly discouraged in production. This opens your system to man-in-the-middle attacks.- Authentication: The document mentions that authentication is initially disabled. Enabling authentication is crucial for security and multi-tenancy.
