Installation von Prometheus mit Grafana in Kubernetes mit Helm Charts

Installation Helm

Als Vorraussetzung muss der Paketmanager Helm in der neuen Verison 3 installiert sein.

yay -S helm

Es kann sein, dass das offiielle Repo mit den Helm Charts noch hinzugefügt werden muss. Das Offizielles Repo (siehe https://helm.sh/docs/intro/quickstart/ ) fügen wir mit

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

zu der Liste der Repositorys hinzu.

Alias für Kubectl

Ich nutze gern k als Alias für Kubectl und verwende es in Artikeln von mir. Wer will, kann folgendes in die .bashrc eintragen:

 alias k=kubectl

Prometheus und Grafana mit Helm 3 installieren

Namespace erstellen

Zunächst erstellen wir einen Namespace für Prometheus und Grafana.

 k create namespace prometheus

Helm 3 Installation durchführen

Nun können Prometheus und Grafana in dem Cluster installiert werden.

 helm install prometheus stable/prometheus-operator --namespace prometheus

Hinweiß: Alternativ kann auch der Parameter –create-namespace mit angegeben werden. So lässt sich ein Schritt einsparen

 helm install prometheus stable/prometheus-operator --namespace prometheus --create-namespace

Den Status des Deployments überprüfen mit

k get pods -n prometheus
NAME                                                     READY   STATUS    RESTARTS   AGE
prometheus-prometheus-node-exporter-cmmbs                1/1     Running   1          38m
prometheus-kube-state-metrics-c65b87574-pn7g2            1/1     Running   1          38m
prometheus-prometheus-node-exporter-6kb7d                1/1     Running   0          29m
prometheus-prometheus-node-exporter-vx7q7                1/1     Running   0          29m
prometheus-prometheus-node-exporter-cct7m                1/1     Running   0          38m
prometheus-prometheus-node-exporter-b4g7j                1/1     Running   1          38m
alertmanager-prometheus-prometheus-oper-alertmanager-0   2/2     Running   0          36m
prometheus-prometheus-node-exporter-kwfs5                1/1     Running   1          38m
prometheus-prometheus-node-exporter-b4smc                1/1     Running   1          38m
prometheus-prometheus-oper-operator-769d757547-mnjh7     2/2     Running   2          38m
prometheus-prometheus-prometheus-oper-prometheus-0       3/3     Running   1          36m
prometheus-prometheus-node-exporter-flk7s                1/1     Running   1          38m
prometheus-grafana-d6545c767-nxz4r                       2/2     Running   0          38m
prometheus-prometheus-node-exporter-85zsp                1/1     Running   1          38m

Die genaue Bezeichnungen der Pods werden später für die Portweiterleitungen benötigt.

Ports öffnen

Nun öffnen wir mit dem Portfoward den Port 9090 für alle Adressen mit:

 k port-forward -n prometheus prometheus-prometheus-prometheus-oper-prometheus-0 --address 0.0.0.0 9090

Nun muss der Port für (Pod Name muss angepasst werden. siehe oben)

 k port-forward -n prometheus prometheus-grafana-d6545c767-5lfqz --address 0.0.0.0 3000

Grafana

Password für Grafana auslesen

 k get secret --namespace prometheus prometheus-grafana -o yaml

Mit OpenSSL dekodieren

 openssl base64 -d
 cHJvbS1vcGVyYXRvcg==
 (STRG + d abschließen)
 prom-operator

 openssl base64 -d
 YWRtaW4=
 admin

Dashboard managen

Dashboard -> Manage

Liste verfügbarer Metriken anzeigen lassen

Port öffnen 8088 auf 8080

k port-forward -n prometheus prometheus-kube-state-metrics-c65b87574-56b58 8088:8080 --address 0.0.0.0

Nun im Browser http://10.10.20.190:8088/metrics öffnen und man erhält folgende Liste

# HELP kube_certificatesigningrequest_labels Kubernetes labels converted to Prometheus labels.
# TYPE kube_certificatesigningrequest_labels gauge
# HELP kube_certificatesigningrequest_created Unix creation timestamp
# TYPE kube_certificatesigningrequest_created gauge
# HELP kube_certificatesigningrequest_condition The number of each certificatesigningrequest condition
# TYPE kube_certificatesigningrequest_condition gauge
# HELP kube_certificatesigningrequest_cert_length Length of the issued cert
# TYPE kube_certificatesigningrequest_cert_length gauge
# HELP kube_configmap_info Information about configmap.
# TYPE kube_configmap_info gauge
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-node-rsrc-use"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-nodes"} 1
kube_configmap_info{namespace="kube-system",configmap="local-path-config"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-prometheus"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-kubelet"} 1
kube_configmap_info{namespace="liferay-prod",configmap="portal-ext.properties"} 1
kube_configmap_info{namespace="kube-system",configmap="traefik-test"} 1
kube_configmap_info{namespace="argocd",configmap="argocd-rbac-cm"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-pod-total"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-apiserver"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-workload"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-prometheus-oper-prometheus-rulefiles-0"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-node"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-workload-total"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-persistentvolumesusage"} 1
kube_configmap_info{namespace="kube-system",configmap="chart-values-traefik"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-pod"} 1
kube_configmap_info{namespace="kube-system",configmap="extension-apiserver-authentication"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-controller-manager"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-cluster-total"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-grafana-test"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-namespace-by-pod"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-scheduler"} 1
kube_configmap_info{namespace="kube-system",configmap="traefik"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-proxy"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-etcd"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-node-cluster-rsrc-use"} 1
kube_configmap_info{namespace="kube-system",configmap="coredns"} 1
kube_configmap_info{namespace="argocd",configmap="argocd-ssh-known-hosts-cm"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-grafana-datasource"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-cluster"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-statefulset"} 1
kube_configmap_info{namespace="argocd",configmap="argocd-cm"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-namespace"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-grafana"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-workloads-namespace"} 1
kube_configmap_info{namespace="argocd",configmap="argocd-tls-certs-cm"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-coredns"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-prometheus-oper-namespace-by-workload"} 1
kube_configmap_info{namespace="liferay-prod",configmap="elasticsearchconfiguration.cfg"} 1
kube_configmap_info{namespace="prometheus",configmap="prometheus-grafana-config-dashboards"} 1
# HELP kube_configmap_created Unix creation timestamp
# TYPE kube_configmap_created gauge
kube_configmap_created{namespace="prometheus",configmap="prometheus-prometheus-oper-statefulset"} 1.597903216e+09
kube_configmap_created{namespace="kube-system",configmap="coredns"} 1.597747102e+09
kube_configmap_created{namespace="argocd",configmap="argocd-ssh-known-hosts-cm"} 1.597747337e+09
kube_configmap_created{namespace="prometheus",configmap="prometheus-prometheus-oper-grafana-datasource"} 1.597903216e+09
kube_configmap_created{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-cluster"} 1.597903216e+09
kube_configmap_created{namespace="prometheus",configmap="prometheus-prometheus-oper-namespace-by-workload"} 1.597903216e+09
kube_configmap_created{namespace="liferay-prod",configmap="elasticsearchconfiguration.cfg"} 1.597834114e+09
kube_configmap_created{namespace="argocd",configmap="argocd-cm"} 1.597747337e+09
kube_configmap_created{namespace="prometheus",configmap="prometheus-prometheus-oper-k8s-resources-namespace"} 1.597903216e+09