zhangguanzhang's Blog

proxmox CSI 使用

proxmox CSI
字数统计: 2.8k阅读时长: 15 min
2026/06/16
loading

记录下 proxmox CSI 部署和使用。

由来

Esxi 虚拟化啥的我个人都不会使用，而是使用 proxmox，以及看到有些人买单台服务器或者电脑部署 k8s 本地和内部开发使用，在数据持久化上没一个好选择，更多的是盲目跟风搞 nfs provider 和 local pv。

昨晚还在纠结用啥 CSI，今天打算部署 local pv了，结果想了下，pve 难道没人做 CSI 吗，调用 pve 的接口，然后创建硬盘挂载给虚机里，虚机内部再给盘格式化文件系统挂载给 pod。这个流程没啥难的，搜了下果然有开源的轮子 sergelogvinov/proxmox-csi-plugin。

部署

开了几台虚机，然后上面部署好 K8S，所有虚拟机都必须设置 SCSI Controller 为 VirtIO SCSI single/VirtIO SCSI 类型才能附加磁盘。

pve auth

由于 CSI controller 需要调用 pve 接口创建磁盘，需要先创建授权信息：

1
2
3
4
5
6
7
8
9
# 创建 role 指定权限
pveum role add CSI -privs "VM.Audit VM.Config.Disk Datastore.Allocate Datastore.AllocateSpace Datastore.Audit"

# 添加用户
pveum user add kubernetes-csi@pve
# role 关联到用户上
pveum aclmod / -user kubernetes-csi@pve -role CSI
# 添加 token
pveum user token add kubernetes-csi@pve csi -privsep 0

上面最后命令会输出一下内容，其中的 full-tokenid 和 value 就是 api 需要的。

1
2
3
4
5
6
7
8
9
┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ kubernetes-csi@pve!csi               │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ e2764307-23f1-4617-xxx-xxx           │
└──────────────┴──────────────────────────────────────┘

节点 label

pve 是可以组集群的，就像虚拟化一样，计算和存储是完全可以分离的，例如 openstack 虚拟化宿主机完全是 cpu/mem 提供者，然后虚机挂载 ceph 的 rbd 块存储当作虚机的硬盘，这样是一种计算存储分离的场景。

考虑到 pve 企业级使用场景存在不一样的计算和存储池子，所以需要给节点对应实际的池子关联上才可以正确处理，这里 Proxmox CSI Kubernetes 拓扑节点标签来定义磁盘位置：

  • topology.kubernetes.io/region: pve 集群名称，如果单机的话需要创建一个集群
  • topology.kubernetes.io/zone: Proxmox 节点名称

可以 pve 上查看：

1
2
root@pve:~# pvecm status
Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?

单机是没有集群的，需要 web 界面上创建后，查看：

1
2
3
4
5
6
7
root@pve:~# pvecm status
Cluster information
-------------------
Name:             pve-cluster
Config Version:   1
Transport:        knet
Secure auth:      on

我这里是单机的，所有虚机都是在同一个 pve 集群的，所以我所有节点都打同样的 label：

1
kubectl label nodes --all topology.kubernetes.io/region=pve-cluster

第二个 label 是 pve 节点名称，可以在 pve 上通过下面命令获取节点信息：

1
2
3
4
5
6
root@pve:~# pvesh get /nodes
┌──────┬────────┬───────┬───────┬────────┬────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────────────────────┬────────────┐
│ node │ status │   cpu │ level │ maxcpu │     maxmem │       mem │ ssl_fingerprint                                                                                 │     uptime │
╞══════╪════════╪═══════╪═══════╪════════╪════════════╪═══════════╪═════════════════════════════════════════════════════════════════════════════════════════════════╪════════════╡
│ pve  │ online │ 0.24% │ s     │    112 │ 503.32 GiB │ 21.83 GiB │ 82:19:EB:63:88:D3:2F:E0:2A:13:BE:F3:60:8D:6F:46:85:16:FD:DD:E4:89:7F:4E:5B:2A:xx:xx:xx:xx:xx:xx │ 23h 7m 11s │
└──────┴────────┴───────┴───────┴────────┴────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┘

我这里是单机的，所有 K8s 虚机都在这个 pve host 上，所以打所有节点：

1
kubectl label nodes --all topology.kubernetes.io/zone=pve

开始安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kubectl create ns csi-proxmox
kubectl label ns csi-proxmox pod-security.kubernetes.io/enforce=privileged

cat > proxmox-csi.yaml << EOF
config:
  clusters:
      # pve api 地址
    - url: https://192.168.1.100:8006/api2/json
      # 由于是自带证书，需要设置为 true
      insecure: true
      token_id: "kubernetes-csi@pve!csi"
      # 前面创建的 token
      token_secret: "e2764307-23f1-4617-xxx-xxx"
      # # pve 集群名字
      region: pve-cluster

# Define storage classes
# See https://pve.proxmox.com/wiki/Storage
storageClass:
  - name: proxmox-data-xfs
    # 这个是 pve web 上使用哪个存储名字，我这里是单独创建了一个给虚机磁盘使用的vmdata
    storage: vmdata
    reclaimPolicy: Delete
    fstype: xfs
    # Define the storage class as default
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
EOF

由于我集群是 k0s 安装的，kubelet dir 不一样，所以尾部加了 helm set 参数

1
2
3
helm upgrade -i -n csi-proxmox -f proxmox-csi.yaml \
  proxmox-csi-plugin oci://ghcr.io/sergelogvinov/charts/proxmox-csi-plugin \
  --set kubeletDir=/var/lib/k0s/kubelet --set node.kubeletDir=/var/lib/k0s/kubelet

查看日志有无错误：

1
2
kubectl -n csi-proxmox get pod -o wide
kubectl -n csi-proxmox logs -l app.kubernetes.io/instance=proxmox-csi-plugin

sc 信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ kubectl get sc
NAME                         PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
proxmox-data-xfs (default)   csi.proxmox.sinextra.dev   Delete          WaitForFirstConsumer   true                   34m
$ kubectl get sc proxmox-data-xfs -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: proxmox-csi-plugin
    meta.helm.sh/release-namespace: csi-proxmox
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2026-06-16T11:55:25Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: proxmox-data-xfs
  resourceVersion: "44789"
  uid: 9c0b8cc0-797c-4c12-9361-3489f235cad0
parameters:
  csi.storage.k8s.io/fstype: xfs
  storage: vmdata
provisioner: csi.proxmox.sinextra.dev
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

测试持久化

pvc 的 storageClassName 和前面设置的要一致：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
cat > pvc-test.yml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: proxmox-csi-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: proxmox-data-xfs
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxmox-csi-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proxmox-csi-test
  template:
    metadata:
      labels:
        app: proxmox-csi-test
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              while true; do
                date >> /data/test.log
                echo "hello from $(hostname)" >> /data/test.log
                sleep 5
              done
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: proxmox-csi-test-pvc
EOF

apply 上面的后，可以看 csi controller 日志创建了盘：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
$ k0s kubectl get pod,pvc,pv -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
pod/proxmox-csi-test-d47b9b59-mtrn2   1/1     Running   0          26s   10.244.2.17   k8s-node2   <none>           <none>

NAME                                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS   AGE   VOLUMEMODE
persistentvolumeclaim/proxmox-csi-test-pvc   Bound    pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa   1Gi        RWO            proxmox-data-xfs   <unset>                 26s   Filesystem

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS       VOLUMEATTRIBUTESCLASS   REASON   AGE   VOLUMEMODE
persistentvolume/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa   1Gi        RWO            Delete           Bound    default/proxmox-csi-test-pvc   proxmox-data-xfs   <unset>                          25s   Filesystem
$ kubectl describe pv pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa 
Name:              pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.proxmox.sinextra.dev
                   volume.kubernetes.io/provisioner-deletion-secret-name: 
                   volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:        [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection external-attacher/csi-proxmox-sinextra-dev]
StorageClass:      proxmox-data-xfs
Status:            Bound
Claim:             default/proxmox-csi-test-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/region in [pve-cluster]
                   topology.kubernetes.io/zone in [pve]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.proxmox.sinextra.dev
    FSType:            xfs
    VolumeHandle:      pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa
    ReadOnly:          false
    VolumeAttributes:      backup=0
                           iothread=1
                           replicate=0
                           storage=vmdata
                           storage.kubernetes.io/csiProvisionerIdentity=1781610926481-2965-csi.proxmox.sinextra.dev
Events:                <none>

$ kubectl -n csi-proxmox logs -l app.kubernetes.io/instance=proxmox-csi-plugin --tail=20
I0616 12:23:33.788519       1 node.go:524] "NodeGetCapabilities: called"
I0616 12:23:33.789271       1 node.go:289] "NodePublishVolume: called" args="{\"publish_context\":{\"DevicePath\":\"/dev/disk/by-id/wwn-0x5056432d49443031\",\"lun\":\"1\"},\"staging_target_path\":\"/var/lib/k0s/kubelet/plugins/kubernetes.io/csi/csi.proxmox.sinextra.dev/2b9e214768e355980134ce053f24fb23c285c4517cac470f687800f6ca252739/globalmount\",\"target_path\":\"/var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"fs_type\":\"xfs\"}},\"volume_context\":{\"backup\":\"0\",\"csi.storage.k8s.io/ephemeral\":\"false\",\"csi.storage.k8s.io/pod.name\":\"proxmox-csi-test-d47b9b59-mtrn2\",\"csi.storage.k8s.io/pod.namespace\":\"default\",\"csi.storage.k8s.io/pod.uid\":\"8d5c2b70-4244-4554-9c2f-e45277601de6\",\"csi.storage.k8s.io/serviceAccount.name\":\"default\",\"iothread\":\"1\",\"replicate\":\"0\",\"storage\":\"vmdata\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1781610926481-2965-csi.proxmox.sinextra.dev\"},\"volume_id\":\"pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa\"}"
I0616 12:23:33.791491       1 mount_linux.go:260] Mounting cmd (mount) with arguments (-t xfs -o bind /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/csi.proxmox.sinextra.dev/2b9e214768e355980134ce053f24fb23c285c4517cac470f687800f6ca252739/globalmount /var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount)
I0616 12:23:33.793022       1 mount_linux.go:260] Mounting cmd (mount) with arguments (-t xfs -o bind,remount,rw /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/csi.proxmox.sinextra.dev/2b9e214768e355980134ce053f24fb23c285c4517cac470f687800f6ca252739/globalmount /var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount)
I0616 12:23:33.794520       1 node.go:383] "NodePublishVolume: volume published for pod" device="/dev/disk/by-id/wwn-0x5056432d49443031" pod="default/proxmox-csi-test-d47b9b59-mtrn2"
I0616 12:23:50.111751       1 node.go:524] "NodeGetCapabilities: called"
I0616 12:23:50.112496       1 node.go:415] "NodeGetVolumeStats: called" args="{\"volume_id\":\"pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa\",\"volume_path\":\"/var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount\"}"
I0616 12:25:05.843077       1 node.go:524] "NodeGetCapabilities: called"
I0616 12:25:05.843850       1 node.go:415] "NodeGetVolumeStats: called" args="{\"volume_id\":\"pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa\",\"volume_path\":\"/var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount\"}"
I0616 12:23:23.922625       1 controller.go:121] "CreateVolume: called" args="{\"accessibility_requirements\":{\"preferred\":[{\"segments\":{\"topology.kubernetes.io/region\":\"pve-cluster\",\"topology.kubernetes.io/zone\":\"pve\"}}],\"requisite\":[{\"segments\":{\"topology.kubernetes.io/region\":\"pve-cluster\",\"topology.kubernetes.io/zone\":\"pve\"}}]},\"capacity_range\":{\"required_bytes\":1073741824},\"name\":\"pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa\",\"parameters\":{\"storage\":\"vmdata\"},\"volume_capabilities\":[{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"fs_type\":\"xfs\"}}]}"
I0616 12:23:23.922689       1 controller.go:143] "CreateVolume: parameters" parameters={"storage":"vmdata","storageFormat":"","backup":false,"iothread":true,"diskIOPS":null,"diskMBps":null,"blockSize":null,"inodeSize":null} modifyParameters={}
I0616 12:23:23.928039       1 controller.go:318] "CreateVolume: creating volume" cluster="pve-cluster" zone="pve" volumeID="pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa" size=1073741824
I0616 12:23:24.794556       1 controller.go:399] "CreateVolume: volume created" cluster="pve-cluster" volumeID="pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa" size=1073741824
I0616 12:23:25.007054       1 controller.go:482] "ControllerPublishVolume: called" args="{\"node_id\":\"k8s-node2\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"fs_type\":\"xfs\"}},\"volume_context\":{\"backup\":\"0\",\"iothread\":\"1\",\"replicate\":\"0\",\"storage\":\"vmdata\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1781610926481-2965-csi.proxmox.sinextra.dev\"},\"volume_id\":\"pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa\"}"
I0616 12:23:25.007093       1 controller.go:523] "ControllerPublishVolume: VM ID not found in NodeID, will lookup by node name" nodeID="k8s-node2"
I0616 12:23:25.012197       1 controller.go:1048] "failed to get proxmox VMID from ProviderID" nodeID="k8s-node2" providerID=""
I0616 12:23:26.377293       1 controller.go:574] "ControllerPublishVolume: volume published" cluster="pve-cluster" volumeID="pve-cluster/pve/vmdata/vm-9999-pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa" nodeID="k8s-node2"

去对应节点上看多了一块盘 sdb：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@k8s-node2:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   250G  0 disk 
├─sda1    8:1    0 249.9G  0 part /
├─sda14   8:14   0     3M  0 part 
└─sda15   8:15   0   124M  0 part /boot/efi
sdb       8:16   0     1G  0 disk /var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount
                                  /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/csi.proxmox.sinextra.dev/2b9e214768e355980134ce053f24fb23c285c4517cac470f687800f6ca252739/globalmount
sr0      11:0    1     4M  0 rom  
root@k8s-node2:~# tail -n 5 /var/lib/k0s/kubelet/pods/8d5c2b70-4244-4554-9c2f-e45277601de6/volumes/kubernetes.io~csi/pvc-5f94f669-2a3c-4cbb-a52f-79a36af40efa/mount/test.log 
hello from proxmox-csi-test-d47b9b59-mtrn2
Tue Jun 16 12:29:24 UTC 2026
hello from proxmox-csi-test-d47b9b59-mtrn2
Tue Jun 16 12:29:29 UTC 2026
hello from proxmox-csi-test-d47b9b59-mtrn2

上面测试完成后删除，因为我 pvc 定义是放在 deploy 同一个 yaml 文件里的，并且 sc 的 reclaimPolicy: Delete，在 delete -f 后真实的硬盘也会被删除掉。

参考

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2026/06/16/proxmox-csi/

发表日期：June 16th 2026, 4:10:30 pm

更新日期：June 16th 2026, 4:10:30 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 由来
  2. 2. 部署
    1. 2.1. pve auth
    2. 2.2. 节点 label
  3. 3. 测试持久化
  4. 4. 参考
Total : 215
2026
2025
2024
2023
2022
2021
2020
2019
2018
2017
2016
diy 8266 ipmi CLP EmuELEC linux k8s Mismatch ingress android Altium Designer N1 openwrt etcd ssl aliyun tun2sock2 ansible consul docker lnmp 安卓 initrd grub raid Linux kernel prometheus exporter boot tcp golang cad centos6 podman centos chromedp chrony cobra kubernetes cni-plugins mips64le clickhouse arm64 kernal kubeconfig cri-dockerd swap adguardhome dns dlv apparmor containerd panic net Dockerfile veth hang e2fsck fsck keepavlied ecs xmrig flannel fio vxlan ftp cifs go mod go gobot esp8266 http gitlab subgroup hello world headscale derper host-gw hostPort iptables ipset inotifywait confd mqtt 原创 jira confluence lvs ipvsadm kube-proxy nodeport kube-controller-manager externalIP hostname ttf woff java springboot 63s timeout vip keepalived fnctl kube-log-runner logrotate kubelet kubeadm operator ConfigMap PodAffinity nfs hotplug kylin systemd Kylin lamp audit fs nginx microsoft graph onedrive nfc pm3 non-root containers coredns gpu nvidia openshift ocp openstack squashfs usb-net overlay /etc/resolv.conf pprof timezone Prometheus cloud-init supervisor pyinstaller python preseed proxmox grpc qemu binfmt_misc loongarch64 cgo redis exsi redhat8.4 rpm rpmbuild salt salt-run ipv6 vmware hung keepalive serviceAccountToken bash shell segfault skopeo boltdb ssh suse dockerhub gcr.io 微信 api ubuntu18 tftp trivy uos wireguard x86 zombie zabbix 正则 awk GetImageBlob sandbox https software openeuler glibc Windows esp loong64 CSI
日志 ipmi 闲搞 kubernetes 心得 kubelet ansible pxe Linux prometheus boot Auto cad 心得 golang raid boot consul kubeconfig grub linux centos6 容器 kickstart docker docker 容器 openstack boot vsftp fstab panic go version gobot http 闲聊 iptables admission kube-controller-manager Prometheus k8s keepalived kubeadm k8s Job vxlan Kylin kmem lua LVM microsoft coredns openwrt raid graph node-cache golang Prometheus prometheus proxmox preseed onedrive 总结 suse CI ubuntu18 docker travis uos wireguard kickstart zombie 运维 技巧