Hubble UI(cilium) 설정하기

CNI Open Source로 cilium을 설치한 후, Add-on 서비스를 구성해본다. 생각보다 이런저런 문제가 있어 기록.

 

 

[참고] cilium 설치 https://dev-tobe.tistory.com/9

kubernetes cluster 수동 설치(w/aws)

 

설치

하단의 링크를 참조하여 간단히 설치 가능.

[참고] https://docs.cilium.io/en/stable/gettingstarted/hubble/#hubble-ui

Service Map & Hubble UI — Cilium 1.15.1 documentation

 

Node간 통신을 위해 TCP 4244 port를 Inbound rule에 추가 필요.

 

[ec2-user@ip-10-180-16-34 ~]$ cilium hubble enable --ui

 

 

Trouble Shoot

설치는 간단하나 설치된 서비스를 점검하다보면 hubble-relay pod가 무한 재기동되는 현상이 있었음

 

Pod Logs(kube-system/hubble-relay-6c5bccdf47-4xjnq:hubble-relay)

level=warning msg="Failed to create peer client for peers synchronization; will try again after the timeout has expired" error="context deadline exceeded" subsys=hubble-relay target="hubble-peer.kube-system.svc.cluster.local:443"

 

 

Configmap Describe(kube-system/hubble-relay-config)

Cluster Domain 이슈로 보이는데, 하단의 peer-service: "hubble-peer.kube-system.svc.cluster.local:443" 를 "hubble-peer:443" 로 변경

[ec2-user@ip-10-180-16-34 ~]$ k describe cm hubble-relay-config -n kube-system
Name:         hubble-relay-config
Namespace:    kube-system
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: cilium
              meta.helm.sh/release-namespace: kube-system

Data
====
config.yaml:
----
cluster-name: kubernetes
peer-service: "hubble-peer.kube-system.svc.cluster.local:443"
listen-address: :4245
gops: true
gops-port: "9893"
dial-timeout: 
retry-timeout: 
sort-buffer-len-max: 
sort-buffer-drain-timeout: 
tls-hubble-client-cert-file: /var/lib/hubble-relay/tls/client.crt
tls-hubble-client-key-file: /var/lib/hubble-relay/tls/client.key
tls-hubble-server-ca-files: /var/lib/hubble-relay/tls/hubble-server-ca.crt
disable-server-tls: true


BinaryData
====

Events:  <none>

 

위의 설명과 같이 조치를 해도 Could not resolve host 오류를 뱉는 경우가 종종 있었는데, kube-system/coredns Deployment를 재기동하면 해결이 된다.(작업을 하다보니 coredns 서비스가 비정상인 경우가 종종 있다. 간단한 서비스 구성 후, 정상적으로 nslookup이 되는지 확인해보는 것이 정신 건강에 이로운 듯)

 

 

 

서비스 정상 확인

[ec2-user@ip-10-180-16-34 ~]$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            hubble-ui          Running: 1
                       cilium             Running: 2
                       cilium-operator    Running: 1
                       hubble-relay       Running: 1
Cluster Pods:          6/6 managed by Cilium
Helm chart version:    1.15.1
Image versions         cilium             quay.io/cilium/cilium:v1.15.1@sha256:351d6685dc6f6ffbcd5451043167cfa8842c6decf80d8c8e426a417c73fb56d4: 2
                       cilium-operator    quay.io/cilium/operator-generic:v1.15.1@sha256:819c7281f5a4f25ee1ce2ec4c76b6fbc69a660c68b7825e9580b1813833fa743: 1
                       hubble-relay       quay.io/cilium/hubble-relay:v1.15.1@sha256:3254aaf85064bc1567e8ce01ad634b6dd269e91858c83be99e47e685d4bb8012: 1
                       hubble-ui          quay.io/cilium/hubble-ui:v0.13.0@sha256:7d663dc16538dd6e29061abd1047013a645e6e69c115e008bee9ea9fef9a6666: 1
                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.13.0@sha256:1e7657d997c5a48253bb8dc91ecee75b63018d16ff5e5797e5af367336bc8803: 1

 

아래와 같이 hubble-ui를 Portforward 설정한 뒤 Local에서 접근하면 서비스 확인이 가능하다.