kubernetes CSI 설치(aws EBS)

[참고] https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

 

Set up driver permissions

Access key 생성

IAM에서 AmazonEBSCSIDriverPolicy를 부여한 User를 간단히 생성

 

k8s secret 생성

[ec2-user@ip-10-180-16-34 tool]$ k create secret generic aws-secret \
>     --namespace kube-system \
>     --from-literal "key_id=${AWS_ACCESS_KEY_ID}" \
>     --from-literal "access_key=${AWS_SECRET_ACCESS_KEY}"
secret/aws-secret created

 

helm 설치

각종 도구의 설치 편의를 도모하기 위해 helm을 설치

[참고] https://helm.sh/docs/intro/install/

Installing Helm

 

[ec2-user@ip-10-180-16-34 tool]$ tar -zxvf helm-v3.14.3-linux-amd64.tar.gz 
linux-amd64/
linux-amd64/LICENSE
linux-amd64/README.md
linux-amd64/helm
[ec2-user@ip-10-180-16-34 tool]$ sudo mv linux-amd64/helm /usr/local/bin/helm
[ec2-user@ip-10-180-16-34 tool]$ helm version
version.BuildInfo{Version:"v3.14.3", GitCommit:"f03cc04caaa8f6d7c3e67cf918929150cf6f3f12", GitTreeState:"clean", GoVersion:"go1.21.7"}

 

 

EBS CSI Driver 설치

[ec2-user@ip-10-180-16-34 tool]$ helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
"aws-ebs-csi-driver" has been added to your repositories
[ec2-user@ip-10-180-16-34 tool]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aws-ebs-csi-driver" chart repository
Update Complete. ⎈Happy Helming!⎈
[ec2-user@ip-10-180-16-34 tool]$ helm upgrade --install aws-ebs-csi-driver \
>     --namespace kube-system \
>     aws-ebs-csi-driver/aws-ebs-csi-driver
Release "aws-ebs-csi-driver" does not exist. Installing it now.
NAME: aws-ebs-csi-driver
LAST DEPLOYED: Tue Mar 19 02:19:11 2024
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
NOTES:
To verify that aws-ebs-csi-driver has started, run:

    kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"

NOTE: The [CSI Snapshotter](https://github.com/kubernetes-csi/external-snapshotter) controller and CRDs will no longer be installed as part of this chart and moving forward will be a prerequisite of using the snap shotting functionality.

WARNING: Upgrading the EBS CSI Driver Helm chart with --reuse-values will no longer be supported in a future release. For more information, see https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1864

 

설치 확인

[ec2-user@ip-10-180-16-34 tool]$ k get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"
NAME                                 READY   STATUS             RESTARTS     AGE
ebs-csi-controller-cc4db67f7-882tw   4/5     CrashLoopBackOff   3 (6s ago)   51s
ebs-csi-controller-cc4db67f7-s9pr7   4/5     CrashLoopBackOff   3 (5s ago)   51s
ebs-csi-node-2w4qw                   2/3     CrashLoopBackOff   3 (9s ago)   51s
ebs-csi-node-cgwqp                   2/3     CrashLoopBackOff   3 (6s ago)   51s

 

 

Issue 해결

EBS 관련 pod 상태가 CrashLoopBackOff로 서비스가 비정상이다. 관련된 pod의 로그는 아래와 같다.

Logs(kube-system/ebs-csi-node-2w4qw)

ebs-plugin I0319 04:58:18.809530       1 node.go:93] "regionFromSession Node service" region=""                             
ebs-plugin I0319 04:58:18.809707       1 metadata.go:85] "retrieving instance data from ec2 metadata"                       
ebs-plugin I0319 04:58:21.935299       1 metadata.go:88] "ec2 metadata is not available"                                    
ebs-plugin I0319 04:58:21.935320       1 metadata.go:96] "retrieving instance data from kubernetes api"                    
ebs-plugin I0319 04:58:21.935818       1 metadata.go:101] "kubernetes api is available"                                     
ebs-plugin panic: node providerID empty, cannot parse                                                                       
ebs-plugin                                                                                                                  
ebs-plugin goroutine 1 [running]:                                                                                           
ebs-plugin github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.newNodeService(0xc000425c00)                            
ebs-plugin     /go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go:96 +0x3b1                           
ebs-plugin github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.NewDriver({0xc000619ec0, 0xd, 0x4?})                    
node-driver-registrar I0319 04:56:46.718155       1 main.go:135] Version: v2.10.0                                           
node-driver-registrar I0319 04:56:46.718276       1 main.go:136] Running node-driver-registrar in mode=                     
node-driver-registrar I0319 04:56:46.718300       1 main.go:157] Attempting to open a gRPC connection with: "/csi/csi.sock" 
node-driver-registrar W0319 04:56:56.719267       1 connection.go:234] Still connecting to unix:///csi/csi.sock

 

관련 문제를 확인할 수 있는 명령어는 아래와 같다.

[ec2-user@ip-10-180-16-34 ~]$ curl -v http://169.254.169.254/latest/meta-data/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Content-Length: 0
< Date: Tue, 19 Mar 2024 05:17:27 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
< 
* Closing connection

 

CLI로 Host의 metadata를 변경할 수 있지만, 아래와 같이 간단하게 Console에서 IMDSv2를 Optional로 변경 및 조치 가능.

 

 

서비스 최종 확인

[ec2-user@ip-10-180-16-34 ~]$ k get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"
NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-79bcbc855c-nr7ck   5/5     Running   0          93s
ebs-csi-controller-79bcbc855c-zm5d4   5/5     Running   0          93s
ebs-csi-node-g4jfv                    3/3     Running   0          82s
ebs-csi-node-x5djn                    3/3     Running   0          82s