菜地里的小菜鸟头像
关注

k8s csi-driver-nfs 动态扩容

1.前置条件​​

  • ​​有一套完整的Kubernetes 集群，且版本 ≥ 1.18（推荐 1.20+）。
  • 已部署好NFS 服务​​：共享目录： /nfs_share/k8s/nfs-csi，NFS服务器：172.16.4.60。

2.部署 NFS CSI 驱动​（csi-driver-nfs）

2.1 克隆官方仓库并进入目录​

CODEBLOCK_PLACEHOLDER_0001 CODEBLOCK_PLACEHOLDER_0002

  • 可以在deploy部署目录下看到csi-driver-nfs最新版本为v4.11.0，我们就使用这个版本，进入到这个目录，yaml文件如下：
[root@master1 ~]# cd /root/statefulset/csi-driver-nfs/deploy/v4.11.0
[root@master1 v4.11.0]# ls
crd-csi-snapshot.yaml    csi-nfs-driverinfo.yaml  csi-snapshot-controller.yaml  rbac-snapshot-controller.yaml  storageclass.yaml
csi-nfs-controller.yaml  csi-nfs-node.yaml        rbac-csi-nfs.yaml             snapshotclass.yaml

2.2 下载镜像

  • 查看csi-driver-nfs v4.11.0版本中需要哪些镜像
[root@master1 v4.11.0]# grep -w image *
csi-nfs-controller.yaml:          image: registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
csi-nfs-controller.yaml:          image: registry.k8s.io/sig-storage/csi-resizer:v1.13.1
csi-nfs-controller.yaml:          image: registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
csi-nfs-controller.yaml:          image: registry.k8s.io/sig-storage/livenessprobe:v2.15.0
csi-nfs-controller.yaml:          image: registry.k8s.io/sig-storage/nfsplugin:v4.11.0
csi-nfs-node.yaml:          image: registry.k8s.io/sig-storage/livenessprobe:v2.15.0
csi-nfs-node.yaml:          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
csi-nfs-node.yaml:          image: registry.k8s.io/sig-storage/nfsplugin:v4.11.0
csi-snapshot-controller.yaml:          image: registry.k8s.io/sig-storage/snapshot-controller:v8.2.0
  • 可以看到csi-nfs-controller.yaml、csi-nfs-node.yaml、csi-snapshot-controller.yaml 文件需要镜像
  • 由于我是内网部署，所以需要在有网的服务器拉取对应镜像后，再推到内网harbor仓库
  • 另外 registry.k8s.io 地址不太好访问，如果有需要的朋友，可以联系我获取
docker pull registry.k8s.io/sig-storage/csi-resizer:v1.13.1
docker pull registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
docker pull registry.k8s.io/sig-storage/livenessprobe:v2.15.0
docker pull registry.k8s.io/sig-storage/nfsplugin:v4.11.0
docker pull registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
docker pull registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
docker pull registry.k8s.io/sig-storage/snapshot-controller:v8.2.0
  • 将下载的镜像保存为tar.gz，上传到内网服务器，并加载镜像
ls *.tar.gz | xargs -i docker load -i {}
  • 修改tag并推送到内网私有harbor，我的harbor仓库地址：172.16.4.177:8090，项目：k8s-csi-nfs
docker tag registry.k8s.io/sig-storage/csi-resizer:v1.13.1 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-resizer:v1.13.1
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-resizer:v1.13.1
docker tag registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
docker tag registry.k8s.io/sig-storage/livenessprobe:v2.15.0 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/livenessprobe:v2.15.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/livenessprobe:v2.15.0
docker tag registry.k8s.io/sig-storage/nfsplugin:v4.11.0 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/nfsplugin:v4.11.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/nfsplugin:v4.11.0
docker tag registry.k8s.io/sig-storage/csi-provisioner:v5.2.0  172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
docker tag registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
docker tag registry.k8s.io/sig-storage/snapshot-controller:v8.2.0  172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/snapshot-controller:v8.2.0
docker push 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/snapshot-controller:v8.2.0

2.3 修改csi-nfs-controller.yaml、csi-nfs-node.yaml、csi-snapshot-controller.yaml 文件中的镜像地址为内网harbor地址

[root@master1 v4.11.0]# grep -w image *
csi-nfs-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
csi-nfs-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-resizer:v1.13.1
csi-nfs-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0
csi-nfs-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/livenessprobe:v2.15.0
csi-nfs-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/nfsplugin:v4.11.0
csi-nfs-node.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/livenessprobe:v2.15.0
csi-nfs-node.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
csi-nfs-node.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/nfsplugin:v4.11.0
csi-snapshot-controller.yaml:          image: 172.16.4.177:8090/k8s-csi-nfs/registry.k8s.io/sig-storage/snapshot-controller:v8.2.0

2.4 部署，依次执行rbac、driverinfo、controller、node四个yaml文件

kubectl apply -f rbac-csi-nfs.yaml
kubectl apply -f csi-nfs-driverinfo.yaml
kubectl apply -f csi-nfs-controller.yaml
kubectl apply -f csi-nfs-node.yaml

2.5 查看pod状态

[root@master1 v4.11.0]# kubectl -n kube-system get pod -o wide -l app=csi-nfs-node
NAME                 READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
csi-nfs-node-4j8r6   3/3     Running   0          82m   172.16.4.89   node3     <none>           <none>
csi-nfs-node-5w28w   3/3     Running   0          82m   172.16.4.86   node1     <none>           <none>
csi-nfs-node-5z4vv   3/3     Running   0          82m   172.16.4.92   master2   <none>           <none>
csi-nfs-node-gcbsn   3/3     Running   0          82m   172.16.4.85   master1   <none>           <none>
csi-nfs-node-hpqvh   3/3     Running   0          82m   172.16.4.90   node4     <none>           <none>
csi-nfs-node-zpsx6   3/3     Running   0          82m   172.16.4.87   node2     <none>           <none>
[root@master1 v4.11.0]# kubectl -n kube-system get pod -o wide -l app=csi-nfs-controller
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
csi-nfs-controller-84f8ff88c5-x8c8b   5/5     Running   0          83m   172.16.4.92   master2   <none>           <none>

3.配置支持动态扩容的 StorageClass​

  • 编辑并修改StorageClass​-nfs.yaml文件，根据实际情况结合注释说明进行修改，直至符合自己的环境
  • storageclass-nfs.yaml: provisioner: nfs.csi.k8s.io 和 csi-nfs-driverinfo.yaml: name: nfs.csi.k8s.io 保持一直
  • allowVolumeExpansion: true 允许动态扩容，必须为true

CODEBLOCK_PLACEHOLDER_0011 CODEBLOCK_PLACEHOLDER_0012

  • 查看sc
[root@master1 csi-driver-nfs]# kubectl get storageclasses.storage.k8s.io
NAME                PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-csi (default)   nfs.csi.k8s.io   Retain          Immediate           true                   69m

4.测试自动创建 PVC和PV

4.1 编辑测试yaml文件

CODEBLOCK_PLACEHOLDER_0014 CODEBLOCK_PLACEHOLDER_0015

4.2 查看pods状态

[root@master1 csi-driver-nfs]# kubectl get pods
NAME             READY   STATUS    RESTARTS        AGE
busybox-test-0   1/1     Running   0               13s
busybox-test-1   1/1     Running   0               9s
busybox-test-2   1/1     Running   0               5s

4.3 查看PVC和PV是否自动创建

  • 通过kubectl可以看到pvc和pv已经自动创建完成
[root@master1 csi-driver-nfs]# kubectl get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-busybox-test-0   Bound    pvc-c4505878-0d72-4955-a5b9-1f678b25b8cf   2Gi        RWO            nfs-csi        44s
data-busybox-test-1   Bound    pvc-c84c5a63-bf69-4e50-ad2b-63193889d236   2Gi        RWO            nfs-csi        40s
data-busybox-test-2   Bound    pvc-ae781199-fb84-4468-9dc8-69791c00793f   2Gi        RWO            nfs-csi        36s

[root@master1 csi-driver-nfs]# kubectl get pv |grep nfs-csi
pvc-ae781199-fb84-4468-9dc8-69791c00793f   2Gi        RWO            Retain           Bound    default/data-busybox-test-2              nfs-csi                      45s
pvc-c4505878-0d72-4955-a5b9-1f678b25b8cf   2Gi        RWO            Retain           Bound    default/data-busybox-test-0              nfs-csi                      53s
pvc-c84c5a63-bf69-4e50-ad2b-63193889d236   2Gi        RWO            Retain           Bound    default/data-busybox-test-1              nfs-csi                      49s
  • 在NFS共享存储上（172.16.4.60）查看是否有pv目录生成
[root@localhost nfs-csi]# ls /nfs_share/k8s/nfs-csi/
pvc-ae781199-fb84-4468-9dc8-69791c00793f  pvc-c4505878-0d72-4955-a5b9-1f678b25b8cf  pvc-c84c5a63-bf69-4e50-ad2b-63193889d236

5.测试动态扩容PVC和PV

  • 确保 nfs-csi StorageClass 的 allowVolumeExpansion 设置为 true
kubectl get storageclass nfs-csi -o jsonpath='{.allowVolumeExpansion}'
# 输出应为 true

5.1 通过【步骤4】可以看到当前启动的busybox-test pod的PVC和PV容量大小为2Gi，也就是占用NFS 2Gi的存储

5.2 触发 PVC 动态扩容​，将 2Gi 扩容到 3Gi

  • 直接编辑PVC请求，将 spec.resources.requests.storage 从 2Gi 改为 3Gi，保存退出

CODEBLOCK_PLACEHOLDER_0020 CODEBLOCK_PLACEHOLDER_0021

  • 使用kubectl patch 命令

CODEBLOCK_PLACEHOLDER_0022 CODEBLOCK_PLACEHOLDER_0023

  • 使用kubectl patch 同时给多个PVC扩容，建议不要一次扩容超过3个以上的PVC，可能会出现问题
#!/bin/bash
for i in {0..2}; do
  kubectl patch pvc data-busybox-test-$i -p '{"spec":{"resources":{"requests":{"storage":"3Gi"}}}}'
done

5.3 查看扩容后的PVC和PV状态

  • 有两个PVC和对应PV已经变成了3Gi
[root@master1 csi-driver-nfs]# kubectl get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-busybox-test-0   Bound    pvc-c4505878-0d72-4955-a5b9-1f678b25b8cf   3Gi        RWO            nfs-csi        27m
data-busybox-test-1   Bound    pvc-c84c5a63-bf69-4e50-ad2b-63193889d236   3Gi        RWO            nfs-csi        27m
data-busybox-test-2   Bound    pvc-ae781199-fb84-4468-9dc8-69791c00793f   2Gi        RWO            nfs-csi        26m
my-mysql-ss-0         Bound    mysql-pv                                   10Gi       RWO                           6d23h
my-s-mysql-s-ss-0     Bound    mysql-s-pv                                 10Gi       RWO                           6d23h

[root@master1 csi-driver-nfs]# kubectl get pv |grep nfs-csi
pvc-ae781199-fb84-4468-9dc8-69791c00793f   2Gi        RWO            Retain           Bound    default/data-busybox-test-2              nfs-csi                      27m
pvc-c4505878-0d72-4955-a5b9-1f678b25b8cf   3Gi        RWO            Retain           Bound    default/data-busybox-test-0              nfs-csi                      27m
pvc-c84c5a63-bf69-4e50-ad2b-63193889d236   3Gi        RWO            Retain           Bound    default/data-busybox-test-1              nfs-csi                      27m

6.参考文档

https://zhuanlan.zhihu.com/p/24321695061

至此就完成了自动创建PVC、PV和动态扩容PVC、PV的操作！！！

转载自 CSDN-专业IT技术社区

原文链接：https://blog.csdn.net/m0_46093787/article/details/163366338

文章来源crawl

相关文章

评论

赞0

评论列表

微信小程序
QQ小程序

关于作者

点赞数：0
关注数：0
粉丝：0
文章：0
关注标签：0
加入于：--