cephfs: use userid and keys for provisioning by black-dragon74 · Pull Request #4988 · ceph/ceph-csi

black-dragon74 · 2024-11-27T13:42:30Z

This patch modifies the code to use userID and
userKey for provisioning of both static and dynamic PVs.

In case user credentials are not found admin credentials are used as a fallback and for backwards compatibility.

black-dragon74 · 2024-11-27T13:51:30Z

Cluster Config

Ceph user

$ ceph auth get client.nick2
[client.nick2]
        key = AQCJHUdnHeDrGBAAd9/9Qc1orCwKwlRZLgsDeQ==
        caps mds = "allow r fsname=myfs path=/volumes, allow rws fsname=myfs path=/volumes/csi"
        caps mgr = "allow rw"
        caps mon = "allow r fsname=myfs"
        caps osd = "allow rw tag cephfs metadata=myfs, allow rw tag cephfs data=myfs"

Provisioner secret

# oc get secrets/rook-csi-cephfs-provisioner-user2 -o yaml
apiVersion: v1
data:
  userID: bmljazI=
  userKey: QVFDSkhVZG5IZURyR0JBQWQ5LzlRYzFvckN3S3dsUlpMZ3NEZVE9PQ==
kind: Secret
metadata:
  creationTimestamp: "2024-11-27T13:27:03Z"
  name: rook-csi-cephfs-provisioner-user2
  namespace: rook-ceph
  resourceVersion: "1722753"
  uid: 88222761-54a2-4eb0-9d2d-9c11326979a8
type: kubernetes.io/rook

Nodestage secret

# oc get secrets/rook-csi-cephfs-node-user2 -o yaml
apiVersion: v1
data:
  userID: bmljazI=
  userKey: QVFDSkhVZG5IZURyR0JBQWQ5LzlRYzFvckN3S3dsUlpMZ3NEZVE9PQ==
kind: Secret
metadata:
  creationTimestamp: "2024-11-27T13:27:03Z"
  name: rook-csi-cephfs-node-user2
  namespace: rook-ceph
  resourceVersion: "1722754"
  uid: 4e9525bd-4854-4cce-9007-58fd261c6c1a
type: kubernetes.io/rook

1. Dynamic PVCs

Resources

❯ oc get sc
NAME          PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-cephfs   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   17m

❯ oc get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cephfs-pvc          Bound    pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced   1Gi        RWO            rook-cephfs    <unset>                 18m

Logs

I1127 13:29:09.069933       1 utils.go:266] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC call: /csi.v1.Controller/CreateVolume
I1127 13:29:09.077837       1 utils.go:267] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced","csi.storage.k8s.io/pvc/name":"cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"rook-ceph","fsName":"myfs","pool":"myfs-replicated"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I1127 13:29:09.170334       1 omap.go:89] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced got omap values: (pool="myfs-metadata", namespace="csi", name="csi.volumes.default"): map[]
I1127 13:29:09.185399       1 omap.go:159] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced set omap keys (pool="myfs-metadata", namespace="csi", name="csi.volumes.default"): map[csi.volume.pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced:595c630d-6e17-4c00-a66e-91785fb01c6d])
I1127 13:29:09.190423       1 omap.go:159] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced set omap keys (pool="myfs-metadata", namespace="csi", name="csi.volume.595c630d-6e17-4c00-a66e-91785fb01c6d"): map[csi.imagename:csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d csi.volname:pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced csi.volume.owner:rook-ceph])
I1127 13:29:09.191264       1 fsjournal.go:318] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced Generated Volume ID (0001-0009-rook-ceph-0000000000000001-595c630d-6e17-4c00-a66e-91785fb01c6d) and subvolume name (csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d) for request name (pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced)
I1127 13:29:09.470449       1 controllerserver.go:475] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced cephfs: successfully created backing volume named csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d for request name pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced
I1127 13:29:09.472306       1 utils.go:273] ID: 108 Req-ID: pvc-39a11e4c-2ddd-46c6-9b5a-6b004bd4eced GRPC response: {"volume":{"capacity_bytes":1073741824,"volume_context":{"clusterID":"rook-ceph","fsName":"myfs","pool":"myfs-replicated","subvolumeName":"csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d","subvolumePath":"/volumes/csi/csi-vol-595c630d-6e17-4c00-a66e-91785fb01c6d/19ea74a6-2409-4220-b930-55deb650dc2a"},"volume_id":"0001-0009-rook-ceph-0000000000000001-595c630d-6e17-4c00-a66e-91785fb01c6d"}}

2. Static PVCs

Resources

❯ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
cephfs-static-pv                           1Gi        RWX            Retain           Bound    rook-ceph/cephfs-static-pvc                  <unset>                          10m

❯ oc get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
cephfs-static-pvc   Bound    cephfs-static-pv                           1Gi        RWX                           <unset>                 10m

Madhu-1 · 2024-12-09T11:15:17Z

charts/ceph-csi-cephfs/README.md


 - For helm 2.x

-    ```bash


It looks like there are more changes than expected in this file related to formatting, Do we need this change?

I could revert the formatting changes. The md files inside charts are using the outdated syntax. Prettier auto formatted them and I decided to stick with it.

What would you suggest?

i would suggest keeping the changes minimal and relevant to the PR as different developers might use different prettier configurations

Madhu-1

small nit, LGTM

charts/ceph-csi-cephfs/values.yaml

iPraveenParihar

LGTM

nixpanic · 2025-01-06T12:29:05Z

@Mergifyio queue

mergify · 2025-01-06T12:29:38Z

queue

🛑 The pull request has been removed from the queue `default`

Details

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

ceph-csi-bot · 2025-01-06T12:30:02Z

/test ci/centos/k8s-e2e-external-storage/1.30

ceph-csi-bot · 2025-01-06T12:30:03Z

/test ci/centos/mini-e2e-helm/k8s-1.30

ceph-csi-bot · 2025-01-06T12:30:03Z

/test ci/centos/mini-e2e/k8s-1.30

ceph-csi-bot · 2025-01-06T12:30:04Z

/test ci/centos/k8s-e2e-external-storage/1.31

ceph-csi-bot · 2025-01-06T12:30:04Z

/test ci/centos/k8s-e2e-external-storage/1.32

ceph-csi-bot · 2025-01-06T12:30:04Z

/test ci/centos/mini-e2e-helm/k8s-1.31

ceph-csi-bot · 2025-01-06T12:30:04Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2025-01-06T12:30:04Z

/test ci/centos/mini-e2e-helm/k8s-1.32

e2e/cephfs_helper.go

Madhu-1 · 2025-01-07T09:27:42Z

/test ci/centos/upgrade-tests-cephfs

black-dragon74 · 2025-01-08T09:10:01Z

@Mergifyio requeue

mergify · 2025-01-08T09:10:08Z

requeue

❌ This pull request head commit has not been previously disembarked from queue.

nixpanic · 2025-01-08T10:12:10Z

@Mergifyio queue

This PR was updated after it was queued, so got unqueued automatically.

mergify · 2025-01-08T10:12:40Z

queue

✅ The pull request has been merged automatically

Details

The pull request has been merged automatically at 7226945

This patch modifies the code to use userID and userKey for provisioning of both static and dynamic PVs. In case user credentials are not found admin credentials are used as a fallback and for backwards compatibility. Signed-off-by: Niraj Yadav <niryadav@redhat.com>

Once the version we use for upgrade testing does not depend on adminID and adminKey we should update the tests to use just the userID and userKey. Signed-off-by: Niraj Yadav <niryadav@redhat.com>

Signed-off-by: Niraj Yadav <niryadav@redhat.com>

ceph-csi-bot · 2025-01-08T11:57:14Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2025-01-08T11:57:15Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2025-01-08T11:57:16Z

/test ci/centos/k8s-e2e-external-storage/1.31

ceph-csi-bot · 2025-01-08T11:57:16Z

/test ci/centos/k8s-e2e-external-storage/1.32

ceph-csi-bot · 2025-01-08T11:57:16Z

/test ci/centos/k8s-e2e-external-storage/1.30

ceph-csi-bot · 2025-01-08T11:57:16Z

/test ci/centos/mini-e2e-helm/k8s-1.32

ceph-csi-bot · 2025-01-08T11:57:16Z

/test ci/centos/mini-e2e-helm/k8s-1.31

ceph-csi-bot · 2025-01-08T11:57:17Z

/test ci/centos/mini-e2e-helm/k8s-1.30

ceph-csi-bot · 2025-01-08T11:57:17Z

/test ci/centos/mini-e2e/k8s-1.32

ceph-csi-bot · 2025-01-08T11:57:17Z

/test ci/centos/mini-e2e/k8s-1.31

ceph-csi-bot · 2025-01-08T11:57:17Z

/test ci/centos/mini-e2e/k8s-1.30

pschichtel · 2025-05-16T23:13:03Z

The change from adminID/adminKey to userID/userKey just took me by surprise after debugging a non-working cephfs resizer for 2 hours. This has been a breaking change in the helm chart, I think this would have deserved a more prominent mention in the release notes.

sshambar · 2025-05-23T02:26:22Z

I just hit this on upgrade to v3.14 too... Issue #4935 makes a suggestion to fallback to userID/userKey if adminID/adminKey not found -- the patch in this pull does the opposite (uses adminID/adminKey if userID/userKey don't exist). Sadly, earlier deployment instructions suggested putting userID/userKey with limited permissions in the same secret for mounting, so it's already there on upgrade to foul up provisioning...

Not sure if it's too late to patch the fallback to reverse the order (as issue suggested), but it would allow error free upgrades...

black-dragon74 marked this pull request as draft November 27, 2024 13:42

mergify bot added the component/cephfs Issues related to CephFS label Nov 27, 2024

black-dragon74 force-pushed the single-set-keys branch 2 times, most recently from 3b74c01 to b48a45a Compare December 3, 2024 08:57

black-dragon74 marked this pull request as ready for review December 3, 2024 10:26

black-dragon74 requested a review from a team December 9, 2024 08:40

Madhu-1 reviewed Dec 9, 2024

View reviewed changes

black-dragon74 force-pushed the single-set-keys branch 2 times, most recently from 8de5147 to b737872 Compare December 9, 2024 12:32

black-dragon74 requested a review from a team December 19, 2024 09:53

Madhu-1 reviewed Jan 3, 2025

View reviewed changes

charts/ceph-csi-cephfs/values.yaml Outdated Show resolved Hide resolved

iPraveenParihar reviewed Jan 6, 2025

View reviewed changes

black-dragon74 force-pushed the single-set-keys branch from b737872 to ef9860d Compare January 6, 2025 10:28

black-dragon74 requested a review from Madhu-1 January 6, 2025 10:29

Madhu-1 previously approved these changes Jan 6, 2025

View reviewed changes

Madhu-1 requested a review from iPraveenParihar January 6, 2025 10:37

iPraveenParihar previously approved these changes Jan 6, 2025

View reviewed changes

black-dragon74 force-pushed the single-set-keys branch from ef9860d to 49effcb Compare January 6, 2025 12:29

mergify bot added the ok-to-test Label to trigger E2E tests label Jan 6, 2025

Madhu-1 reviewed Jan 7, 2025

View reviewed changes

e2e/cephfs_helper.go Show resolved Hide resolved

black-dragon74 commented Jan 7, 2025

View reviewed changes

e2e/cephfs_helper.go Show resolved Hide resolved

Madhu-1 approved these changes Jan 7, 2025

View reviewed changes

iPraveenParihar approved these changes Jan 8, 2025

View reviewed changes

black-dragon74 added 4 commits January 8, 2025 11:56

e2e: Note to update to userID and userKey

08874aa

Once the version we use for upgrade testing does not depend on adminID and adminKey we should update the tests to use just the userID and userKey. Signed-off-by: Niraj Yadav <niryadav@redhat.com>

helm: Update helm charts to use userID and Keys

b9aa0ce

Signed-off-by: Niraj Yadav <niryadav@redhat.com>

doc: update documentation for userID and userKey

d2ec4e2

Signed-off-by: Niraj Yadav <niryadav@redhat.com>

black-dragon74 force-pushed the single-set-keys branch from bcb31e5 to d2ec4e2 Compare January 8, 2025 11:56

mergify bot added the ok-to-test Label to trigger E2E tests label Jan 8, 2025

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Jan 8, 2025

mergify bot merged commit 7226945 into ceph:devel Jan 8, 2025
37 checks passed

Conversation

black-dragon74 commented Nov 27, 2024 • edited by iPraveenParihar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

black-dragon74 commented Nov 27, 2024

Cluster Config

Ceph user

Provisioner secret

Nodestage secret

1. Dynamic PVCs

Resources

Logs

2. Static PVCs

Resources

Uh oh!

Madhu-1 Dec 9, 2024

Choose a reason for hiding this comment

Uh oh!

black-dragon74 Dec 9, 2024

Choose a reason for hiding this comment

Uh oh!

Madhu-1 Dec 9, 2024

Choose a reason for hiding this comment

Uh oh!

black-dragon74 Dec 9, 2024

Choose a reason for hiding this comment

Uh oh!

Madhu-1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iPraveenParihar left a comment

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Jan 6, 2025

Uh oh!

mergify bot commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request has been removed from the queue default

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

ceph-csi-bot commented Jan 6, 2025

Uh oh!

Uh oh!

Uh oh!

Madhu-1 commented Jan 7, 2025

Uh oh!

black-dragon74 commented Jan 8, 2025

Uh oh!

mergify bot commented Jan 8, 2025

❌ This pull request head commit has not been previously disembarked from queue.

Uh oh!

nixpanic commented Jan 8, 2025

Uh oh!

mergify bot commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ The pull request has been merged automatically

Uh oh!

ceph-csi-bot commented Jan 8, 2025

Uh oh!

ceph-csi-bot commented Jan 8, 2025

Uh oh!

ceph-csi-bot commented Jan 8, 2025

Uh oh!

ceph-csi-bot commented Jan 8, 2025

Uh oh!

ceph-csi-bot commented Jan 8, 2025

black-dragon74 commented Nov 27, 2024 •

edited by iPraveenParihar

Loading

mergify bot commented Jan 6, 2025 •

edited

Loading

🛑 The pull request has been removed from the queue `default`

mergify bot commented Jan 8, 2025 •

edited

Loading