Skip to content

Commit dfba12e

Browse files
authored
Merge of #5829
2 parents 80c0474 + ea99e05 commit dfba12e

File tree

19 files changed

+415
-49
lines changed

19 files changed

+415
-49
lines changed

PendingReleaseNotes.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,6 @@
44

55
## Features
66

7+
- nfs: allow changing NFS-server through ControllerModifyVolume [PR](https://github.com/ceph/ceph-csi/pull/5829)
8+
79
## NOTE

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,7 @@ for its support details.
114114
| | Creating and deleting snapshot | Alpha | >= v3.7.0 | >= v1.1.0 | Pacific (>=v16.2.0) | >= v1.17.0 |
115115
| | Provision volume from snapshot | Alpha | >= v3.7.0 | >= v1.1.0 | Pacific (>=v16.2.0) | >= v1.17.0 |
116116
| | Provision volume from another volume | Alpha | >= v3.7.0 | >= v1.1.0 | Pacific (>=v16.2.0) | >= v1.16.0 |
117+
| | Modify volume parameters with ControllerModifyVolume | Alpha | >= v3.17.0 | >= v1.12.0 | Pacific (>=v16.2.0) | >= v1.34.0 |
117118

118119
`NOTE`: The `Alpha` status reflects possible non-backward
119120
compatible changes in the future, and is thus not recommended

build.env

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,11 @@ ROOK_VERSION=v1.16.4
6161
ROOK_CEPH_CLUSTER_IMAGE=quay.io/ceph/ceph:v19.2.2
6262

6363
# CSI sidecar version
64+
K8S_IMAGE_REPO=gcr.io/k8s-staging-sig-storage
6465
CSI_ATTACHER_VERSION=v4.10.0
6566
CSI_SNAPSHOTTER_VERSION=v8.4.0
66-
CSI_RESIZER_VERSION=v2.0.0
67-
CSI_PROVISIONER_VERSION=v6.0.0
67+
CSI_RESIZER_VERSION=canary
68+
CSI_PROVISIONER_VERSION=canary
6869
CSI_NODE_DRIVER_REGISTRAR_VERSION=v2.15.0
6970

7071
# e2e settings

deploy/nfs/kubernetes/csi-nfsplugin.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,8 @@ spec:
7373
mountPath: /etc/ceph/
7474
- name: ceph-csi-config
7575
mountPath: /etc/ceph-csi-config/
76+
- name: keys-tmp-dir
77+
mountPath: /tmp/csi/keys
7678
- name: driver-registrar
7779
# This is necessary only for systems with SELinux, where
7880
# non-privileged sidecar containers cannot access unix domain socket
@@ -133,3 +135,7 @@ spec:
133135
- name: ceph-csi-config
134136
configMap:
135137
name: ceph-csi-config
138+
- name: keys-tmp-dir
139+
emptyDir: {
140+
medium: "Memory"
141+
}

deploy/nfs/kubernetes/csi-provisioner-rbac.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,9 @@ rules:
6060
- apiGroups: ["snapshot.storage.k8s.io"]
6161
resources: ["volumesnapshots"]
6262
verbs: ["get", "list"]
63+
- apiGroups: ["storage.k8s.io"]
64+
resources: ["volumeattributesclasses"]
65+
verbs: ["get", "list"]
6366

6467
---
6568
kind: ClusterRoleBinding

docs/volumeattributesclass.md

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# VolumeAttributesClass for Volume Modification
2+
3+
Kubernetes offers a method for modifying Volume parameters after they have been
4+
created. This is done through VolumeAttributesClasses and is described in a
5+
[blog
6+
post](https://kubernetes.io/blog/2025/09/08/kubernetes-v1-34-volume-attributes-class).
7+
8+
## Prerequisites
9+
10+
- Kubernetes 1.34 is the first release where support for
11+
VolumeAttributesClasses (the `ControllerModifyVolume` CSI procedure) is GA.
12+
Older versions of Kubernetes may not work reliably.
13+
- The Kubernetes CSI external-provisioner (`csi-provisioner` sidecar) release
14+
needs to be higher or equal to v6.1 (support for secrets).
15+
- The Kubernetes CSI external-resizer (`csi-resizer` sidecar) release
16+
needs to be higher or equal to v2.1 (support for secrets).
17+
18+
## Secret references in the StorageClass
19+
20+
When setting a VolumeAttributesClass on a PersistentVolumeClaim, the
21+
`ControllerModifyVolume` CSI procedure is called on the provisioner. This
22+
procedure needs secrets (that contain the Ceph credentials) in order to
23+
communicate with the Ceph cluster. Below are the keys that should be set in the
24+
StorageClass' `parameters`:
25+
26+
- `csi.storage.k8s.io/controller-modify-secret-name`
27+
- `csi.storage.k8s.io/controller-modify-secret-namespace`
28+
29+
In addition to the execution of `ControllerModifyVolume`, the Node-plugin needs
30+
access to the Volume on the Ceph cluster to fetch the updated parameters.
31+
Usually NFS and NVMe-oF do not need any Ceph credentials for the Node-plugin,
32+
but for using VolumeAttributesClasses to modify Volumes this is a requirement.
33+
34+
Depending on the storage backend, Volumes are optionally _Staged_ before
35+
getting _Published_. NVMe-oF uses the staging process, and needs credentials
36+
there, hence these parameters should be set:
37+
38+
- `csi.storage.k8s.io/node-stage-secret-name`
39+
- `csi.storage.k8s.io/node-stage-secret-namespace`
40+
41+
NFS does not use the staging process, and only needs the credentials during the
42+
publishing process:
43+
44+
- `csi.storage.k8s.io/node-publish-secret-name`
45+
- `csi.storage.k8s.io/node-publish-secret-namespace`
46+
47+
## Adding support to existing Volumes
48+
49+
PersistentVolumeClaims that have not been created with the right secret
50+
references in the StorageClass will not be modifiable with a
51+
VolumeAttributesClass without manual intervention.
52+
53+
In order to be able to modify parameters with a VolumeAttributesClass,
54+
annotations should be added to the PersistentVolume that is _Bound_ to the
55+
PersistentVolumeClaim. These annotations should refer to the namespace and
56+
secret where the credentials are available (just like the namespace and secret
57+
that would be referenced in the StorageClass).
58+
59+
- `volume.kubernetes.io/controller-modify-secret-name`
60+
- `volume.kubernetes.io/controller-modify-secret-namespace`
61+
62+
Note that these annotations only make it possible for the provisioner to modify
63+
parameters. This does not allow the Node-plugin accessing the Ceph cluster to
64+
fetch updated parameters when the Volume is _staged_ or _published_. The
65+
secrets for staging and publishing can not (easily) be updated after the fact,
66+
these are part of the fixed parameters in the PersistentVolume.

e2e/go.mod

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ require (
1818
// when updating k8s.io modules, update the 'replace' section below too
1919
k8s.io/api v0.35.0
2020
k8s.io/apimachinery v0.35.0
21-
k8s.io/client-go v12.0.0+incompatible
21+
k8s.io/client-go v0.35.0
2222
k8s.io/cloud-provider v0.35.0
2323
k8s.io/kubernetes v1.35.0
2424
k8s.io/pod-security-admission v0.35.0
@@ -36,6 +36,8 @@ replace (
3636
k8s.io/kubelet => k8s.io/kubelet v0.35.0
3737
)
3838

39+
exclude k8s.io/client-go v12.0.0+incompatible
40+
3941
require (
4042
cel.dev/expr v0.24.0 // indirect
4143
cyphar.com/go-pathrs v0.2.1 // indirect
@@ -123,7 +125,7 @@ require (
123125
k8s.io/klog/v2 v2.130.1 // indirect
124126
k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
125127
k8s.io/kubectl v0.0.0 // indirect
126-
k8s.io/kubelet v0.33.2 // indirect
128+
k8s.io/kubelet v0.0.0 // indirect
127129
k8s.io/mount-utils v0.35.0 // indirect
128130
k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // indirect
129131
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 // indirect

e2e/nfs.go

Lines changed: 129 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,8 @@ func createNFSStorageClass(
178178
sc.Parameters["csi.storage.k8s.io/controller-expand-secret-namespace"] = cephCSINamespace
179179
sc.Parameters["csi.storage.k8s.io/controller-expand-secret-name"] = cephFSProvisionerSecretName
180180

181-
sc.Parameters["csi.storage.k8s.io/node-stage-secret-namespace"] = cephCSINamespace
182-
sc.Parameters["csi.storage.k8s.io/node-stage-secret-name"] = cephFSNodePluginSecretName
181+
sc.Parameters["csi.storage.k8s.io/node-publish-secret-namespace"] = cephCSINamespace
182+
sc.Parameters["csi.storage.k8s.io/node-publish-secret-name"] = cephFSNodePluginSecretName
183183

184184
if enablePool {
185185
sc.Parameters["pool"] = "myfs-replicated"
@@ -227,6 +227,79 @@ func createNFSStorageClass(
227227
})
228228
}
229229

230+
func createNFSVolumeAttributesClass(
231+
c clientset.Interface,
232+
f *framework.Framework,
233+
params map[string]string,
234+
) error {
235+
vacPath := fmt.Sprintf("%s/%s", nfsExamplePath, "volumeattributeclass.yaml")
236+
vac, err := getVolumeAttributesClass(vacPath)
237+
if err != nil {
238+
return err
239+
}
240+
241+
// overload any parameters that were passed
242+
if params == nil {
243+
// create an empty params, so that params["clusterID"] below
244+
// does not panic
245+
params = map[string]string{}
246+
}
247+
for param, value := range params {
248+
vac.Parameters[param] = value
249+
}
250+
251+
vac.DriverName = nfsDriverName
252+
253+
timeout := time.Duration(deployTimeout) * time.Minute
254+
255+
return wait.PollUntilContextTimeout(context.TODO(), poll, timeout, true, func(ctx context.Context) (bool, error) {
256+
_, err = c.StorageV1().VolumeAttributesClasses().Create(ctx, &vac, metav1.CreateOptions{})
257+
if err != nil {
258+
framework.Logf("error creating VolumeAttributesClass %q: %v", vac.Name, err)
259+
if apierrs.IsAlreadyExists(err) {
260+
return true, nil
261+
}
262+
if isRetryableAPIError(err) {
263+
return false, nil
264+
}
265+
266+
return false, fmt.Errorf("failed to create VolumeAttributesClass %q: %w", vac.Name, err)
267+
}
268+
269+
return true, nil
270+
})
271+
}
272+
273+
func deleteNFSVolumeAttributesClass(
274+
c clientset.Interface,
275+
f *framework.Framework,
276+
) error {
277+
vacPath := fmt.Sprintf("%s/%s", nfsExamplePath, "vac-relocated.yaml")
278+
vac, err := getVolumeAttributesClass(vacPath)
279+
if err != nil {
280+
return err
281+
}
282+
283+
timeout := time.Duration(deployTimeout) * time.Minute
284+
285+
return wait.PollUntilContextTimeout(context.TODO(), poll, timeout, true, func(ctx context.Context) (bool, error) {
286+
err = c.StorageV1().VolumeAttributesClasses().Delete(ctx, vac.Name, metav1.DeleteOptions{})
287+
if err != nil {
288+
framework.Logf("error deleting VolumeAttributesClass %q: %v", vac.Name, err)
289+
if apierrs.IsNotFound(err) {
290+
return true, nil
291+
}
292+
if isRetryableAPIError(err) {
293+
return false, nil
294+
}
295+
296+
return false, fmt.Errorf("failed to delete VolumeAttributesClass %q: %w", vac.Name, err)
297+
}
298+
299+
return true, nil
300+
})
301+
}
302+
230303
// unmountNFSVolume unmounts a NFS volume mounted on a pod.
231304
func unmountNFSVolume(f *framework.Framework, appName, pvcName string) error {
232305
pod, err := f.ClientSet.CoreV1().Pods(f.UniqueName).Get(context.TODO(), appName, metav1.GetOptions{})
@@ -500,6 +573,60 @@ var _ = Describe("nfs", func() {
500573
}
501574
})
502575

576+
By("create a storageclass with relocated server and a PVC then bind it to an app", func() {
577+
if !k8sVersionGreaterEquals(c, 1, 34) {
578+
framework.Logf("skipping VolumeAttributesClass test, needs Kubernetes >= 1.34")
579+
580+
return
581+
}
582+
583+
err := createNFSStorageClass(f.ClientSet, f, false, map[string]string{
584+
"server": "relocated.example.net", // mounting will fail without vac
585+
})
586+
if err != nil {
587+
logAndFail("failed to create NFS storageclass: %v", err)
588+
}
589+
err = createNFSVolumeAttributesClass(f.ClientSet, f, map[string]string{
590+
"server": "rook-ceph-nfs-my-nfs-a." + rookNamespace + ".svc.cluster.local",
591+
})
592+
if err != nil {
593+
logAndFail("failed to create NFS voluemattributesclass: %v", err)
594+
}
595+
596+
pvc, err := loadPVC(pvcPath)
597+
if err != nil {
598+
logAndFail("Could not create PVC: 1 %v", err)
599+
}
600+
pvc.Namespace = f.UniqueName
601+
vacName := "updated-parameters"
602+
pvc.Spec.VolumeAttributesClassName = &vacName
603+
604+
app, err := loadApp(appPath)
605+
if err != nil {
606+
logAndFail("failed to load application: %v", err)
607+
}
608+
app.Namespace = f.UniqueName
609+
app.Spec.Volumes[0].PersistentVolumeClaim.ClaimName = pvc.Name
610+
err = createPVCAndApp("", f, pvc, app, deployTimeout)
611+
if err != nil {
612+
logAndFail("failed to create PVC or application: %v", err)
613+
}
614+
615+
// delete PVC and app
616+
err = deletePVCAndApp("", f, pvc, app)
617+
if err != nil {
618+
logAndFail("failed to delete PVC or application: %v", err)
619+
}
620+
err = deleteResource(nfsExamplePath + "storageclass.yaml")
621+
if err != nil {
622+
logAndFail("failed to delete NFS storageclass: %v", err)
623+
}
624+
err = deleteNFSVolumeAttributesClass(f.ClientSet, f)
625+
if err != nil {
626+
logAndFail("failed to delete NFS voluemattributesclass: %v", err)
627+
}
628+
})
629+
503630
By("create a storageclass with sys,krb5i security and a PVC then bind it to an app", func() {
504631
err := createNFSStorageClass(f.ClientSet, f, false, map[string]string{
505632
"secTypes": "sys,krb5i",

e2e/utils.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,13 @@ func getStorageClass(path string) (scv1.StorageClass, error) {
369369
return sc, err
370370
}
371371

372+
func getVolumeAttributesClass(path string) (scv1.VolumeAttributesClass, error) {
373+
vac := scv1.VolumeAttributesClass{}
374+
err := unmarshal(path, &vac)
375+
376+
return vac, err
377+
}
378+
372379
func getSecret(path string) (v1.Secret, error) {
373380
sc := v1.Secret{}
374381
err := unmarshal(path, &sc)

e2e/vendor/modules.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ github.com/ceph/ceph-csi/internal/util/stripsecrets
3131
github.com/ceph/ceph-csi/pkg/util/crypto
3232
github.com/ceph/ceph-csi/pkg/util/kernel
3333
# github.com/ceph/ceph-csi/api v0.0.0-00010101000000-000000000000 => ../api
34-
## explicit; go 1.24.0
34+
## explicit; go 1.25.0
3535
github.com/ceph/ceph-csi/api/deploy/kubernetes
3636
# github.com/cespare/xxhash/v2 v2.3.0
3737
## explicit; go 1.11
@@ -757,7 +757,7 @@ k8s.io/apiserver/pkg/util/feature
757757
k8s.io/apiserver/pkg/util/webhook
758758
k8s.io/apiserver/pkg/util/x509metrics
759759
k8s.io/apiserver/pkg/warning
760-
# k8s.io/client-go v12.0.0+incompatible => k8s.io/client-go v0.35.0
760+
# k8s.io/client-go v0.35.0 => k8s.io/client-go v0.35.0
761761
## explicit; go 1.25.0
762762
k8s.io/client-go/applyconfigurations/admissionregistration/v1
763763
k8s.io/client-go/applyconfigurations/admissionregistration/v1alpha1
@@ -1110,7 +1110,7 @@ k8s.io/kube-openapi/pkg/validation/strfmt/bson
11101110
## explicit; go 1.25.0
11111111
k8s.io/kubectl/pkg/scale
11121112
k8s.io/kubectl/pkg/util/podutils
1113-
# k8s.io/kubelet v0.33.2 => k8s.io/kubelet v0.35.0
1113+
# k8s.io/kubelet v0.0.0 => k8s.io/kubelet v0.35.0
11141114
## explicit; go 1.25.0
11151115
k8s.io/kubelet/pkg/apis
11161116
k8s.io/kubelet/pkg/apis/stats/v1alpha1

0 commit comments

Comments
 (0)