• k8s 1.15.0
• crio 1.15.0
• flannel 0.10.0
• ceph 14.2.1(客户端/服务端)

• cephcsi v1.1.0

rbd-csi

1. missing ID field 'userID' in secrets

现象

能成功创建pvc且状态是Bound状态，但是通过kubectl delete -f rbd/pvc.yaml 后相关联的pv没有一并删除，需要手动删除；且创建的pod一直处于ContainerCreating状态

[root@node-130-11 ~]# kubectl get pvc

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE

rbd-pvc   Bound    pvc-c688dbfb-f00f-48a6-852e-0fc9a17b00e9   3Gi        RWO            csi-rbd-sc     4m57s

# kubectl describe pod csi-rbd-demo-pod 报如下错

  Warning  FailedMount             7m25s (x18 over 51m)  kubelet, 192.168.130.13  Unable to mount volumes for pod "csi-rbd-demo-pod_default(0ab337ea-9a9d-4781-b668-aa152488cd10)": timeout expired waiting for volumes to attach or mount for pod "default"/"csi-rbd-demo-pod". list of unmounted volumes=[mypvc]. list of unattached volumes=[mypvc default-token-5mxg9]

  Warning  FailedMount             48s (x31 over 53m)    kubelet, 192.168.130.13  MountVolume.SetUp failed for volume "pvc-d9814028-805f-4ac0-92a6-e7ed1611dd51" : rpc error: code = Internal desc = missing ID field 'userID' in secrets

# 查看csi-rbdplugin日志，报错如下

14m         Warning   ProvisioningFailed        persistentvolumeclaim/rbd-pvc                               failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = error (an error (exit status 1) occurred while running rados args: [-m 192.168.130.14,192.168.130.15,192.168.130.16 --id  --key=***stripped*** -c /etc/ceph/ceph.conf -p rbd getomapval csi.volumes.default csi.volume.pvc-81dcd4ad-21af-4afb-af9b-ca3657c25824 /tmp/omap-get-263810844]) occurred, command output streams is ( 2019-08-01 16:36:34.257 7f12c1fb7880 -1 auth: unable to find a keyring on /etc/ceph/ceph.client..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

2019-08-01 16:36:34.258 7f12c1fb7880 -1 auth: unable to find a keyring on /etc/ceph/ceph.client..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

2019-08-01 16:36:34.258 7f12c1fb7880 -1 auth: unable to find a keyring on /etc/ceph/ceph.client..keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

2019-08-01 16:36:34.261 7f12b1943700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

2019-08-01 16:36:34.264 7f12b2945700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]

failed to fetch mon config (--no-mon-config to skip)

解决问题相关issue
https://github.com/ceph/ceph-csi/pull/395
https://github.com/rook/rook/issues/3357
https://github.com/rook/rook/issues/3168
https://github.com/ceph/ceph-csi/issues/344

翻遍了rook, ceph-csi的所有相关issue，只找到一个非常相似的 https://github.com/rook/rook/issues/3357 ，但只提到Stringdata, data 是否base64编码问题，确认与本场景无关，无赖之下，只能最后尝试下非稳定版的 quay.io/cephcsi/cephcsi:canary 镜像，之后问题全部得以解决，pod成功运行，且删除pvc后，pv也会自动回收，完美。