Prometheus 설치 (kubernetes/aws)

helm을 사용하여 kubernetest cluster에 monitoring 환경을 구성한다.

 

[참고] https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

 

helm repository 설정

aws-ebs-csi-driver는 https://dev-tobe.tistory.com/12 과정에서 설치했으므로 생략.

[ec2-user@ip-10-180-16-34 ~]$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
[ec2-user@ip-10-180-16-34 ~]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aws-ebs-csi-driver" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈

 

설치 최적화

설치하려는 namespace(monitoring)를 생성하고, Storage Class도 추가한다. 비용도 그렇고 gp3 type을 활용.

[ec2-user@ip-10-180-16-34 ~]$ k create namespace monitoring
namespace/monitoring created
[ec2-user@ip-10-180-16-34 ~]$ EBS_AZ=$(kubectl get nodes \
>   -o=jsonpath="{.items[0].metadata.labels['topology\.kubernetes\.io\/zone']}")
[ec2-user@ip-10-180-16-34 ~]$ echo "
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
>   name: prometheus
>   namespace: monitoring
> provisioner: ebs.csi.aws.com
> parameters:
>   type: gp3
> reclaimPolicy: Retain
> allowedTopologies:
> - matchLabelExpressions:
>   - key: topology.ebs.csi.aws.com/zone
>     values:
>     - $EBS_AZ
> " | kubectl apply -f -
storageclass.storage.k8s.io/prometheus created

 

value.yaml 수정

앞에 생성한 Storage Class의 적용과 Storage Size 축소, 그 외 현재 불필요한 alertmanager 설치 제외를 위한 수정을 진행한다.

[ec2-user@ip-10-180-16-34 ~]$ helm show values prometheus-community/prometheus >> prometheus_value.yaml

 

prometheus_value.yaml 수정 사항

##AS-IS
..생략..
    ## Prometheus server data Persistent Volume size
    ##
    size: 8Gi
..생략..    
    # storageClass: "-"
..생략..
alertmanager:
  ## If false, alertmanager will not be installed
  ##
  enabled: true


##TO-BE
..생략..
    ## Prometheus server data Persistent Volume size
    ##
    size: 2Gi
..생략..    
    storageClass: "prometheus"
..생략..
alertmanager:
  ## If false, alertmanager will not be installed
  ##
  enabled: false

 

설치

helm을 통한 설치는 역시나 간단하게 완료. 추가로 최적화할 부분이 있다면 앞 단계에서 보완 필요.

[ec2-user@ip-10-180-16-34 ~]$ helm install -f prometheus_value.yaml prometheus-community/prometheus -n monitoring --generate-name
NAME: prometheus-1710903430
LAST DEPLOYED: Wed Mar 20 02:57:12 2024
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-1710903430-server.monitoring.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus-1710903430" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090

..생략..

The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-1710903430-prometheus-pushgateway.monitoring.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

 

 

설치 확인

prometheus 관련 pod들이 정상적으로 기동되었는지와 Storage가 정상적으로 mount되었는지 확인

[ec2-user@ip-10-180-16-34 ~]$ k get po -n monitoring
NAME                                                            READY   STATUS    RESTARTS   AGE
prometheus-1710903430-kube-state-metrics-6957ff6d4c-q894p       1/1     Running   0          2m37s
prometheus-1710903430-prometheus-node-exporter-dxm9s            1/1     Running   0          2m37s
prometheus-1710903430-prometheus-node-exporter-vnpk5            1/1     Running   0          2m37s
prometheus-1710903430-prometheus-pushgateway-7978f97979-b2vn6   1/1     Running   0          2m37s
prometheus-1710903430-server-5d8d6566bb-bv2s5                   2/2     Running   0          2m37s

[ec2-user@ip-10-180-16-34 ~]$ k get pv -n monitoirng
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-8f6c602c-1ac5-44a0-81ca-c5ea4a229c6e   2Gi        RWO            Retain           Bound    monitoring/prometheus-1710903430-server   prometheus     <unset>                          2m41s

 

서비스 확인

kubectl port-forward를 활용하여 로컬에서 Prometheus 서비스에 접근한다.

[ec2-user@ip-10-180-16-34 ~]$ k port-forward svc/prometheus-1710903430-server 9090:80 -n monitoring
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

간단한 Query를 실행하여 정상적으로 서비스가 동작하는지 아래와 같이 확인.