Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The user can fail to write to a volume even if fsGroup is properly set in securityContext #835

Open
mpatlasov opened this issue Aug 30, 2024 · 3 comments · May be fixed by #848
Open

The user can fail to write to a volume even if fsGroup is properly set in securityContext #835

mpatlasov opened this issue Aug 30, 2024 · 3 comments · May be fixed by #848

Comments

@mpatlasov
Copy link
Contributor

What happened:

Even if user explicitly requires the same fsGroup as runAsUser and runAsGroup for a Pod:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002

he/she will fail to write to the volume:

bash-5.2$ touch /mnt/claim/FILE
touch: cannot touch '/mnt/claim/FILE': Permission denied

if StorageClass does not set uid/gid or dir_mode or noperm:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: samba
provisioner: smb.csi.k8s.io
parameters:
  source: //samba-server.samba-server.svc.cluster.local/share
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: samba-server
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: samba-server
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - file_mode=0777
  - mfsymlinks
  - cache=strict
  - noserverino

What you expected to happen:

If user requests the same fsGroup as runAsGroup in securityContext section of a pod definition, SMB CSI driver must construct mount options letting full access for that volume.

How to reproduce it:

Create StorageClass:

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: samba
provisioner: smb.csi.k8s.io
parameters:
  source: //samba-server.samba-server.svc.cluster.local/share
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: samba-server
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: samba-server
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - file_mode=0777
  - mfsymlinks
  - cache=strict
  - noserverino
EOF

for a samba server from https:/samba-in-kubernetes/samba-container .

Create a Pod requesting samba PV:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi
  storageClassName: samba
---
apiVersion: v1
kind: Pod
metadata:
  name: fedora
  labels:
    app: fedora
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1002
    runAsGroup: 1002
    fsGroup: 1002
  volumes:
  - name: claim2-vol
    persistentVolumeClaim:
      claimName: claim
  containers:
  - image: registry.fedoraproject.org/fedora-minimal
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: fedora
    securityContext:
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault
      capabilities:
        drop:
        - "ALL"
    volumeMounts:
      - mountPath: "/mnt/claim"
        name: claim2-vol
  restartPolicy: Always

Verify that the volume is not write-able from the Pod:

$ kubectl exec -it fedora -- bash
bash-5.2$ touch /mnt/claim/FILE
touch: cannot touch '/mnt/claim/FILE': Permission denied

Anything else we need to know?:

Environment:

  • CSI Driver version: Image manually built by docker build ... -f ./cmd/smbplugin/Dockerfile . from the top of upstream master branch (commit 4d10e97 of https:/kubernetes-csi/csi-driver-smb.git)

  • Kubernetes version (use kubectl version):

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3
@andyzhangx
Copy link
Member

@mpatlasov can you provide the csi driver logs on the node? kubelet should Delegate fsGroup to CSI Driver in NodeStageVolume, thus you don't need to set gid in storage class:

https://kubernetes-csi.github.io/docs/support-fsgroup.html#delegate-fsgroup-to-csi-driver

@mpatlasov
Copy link
Contributor Author

@andyzhangx , csi driver uses Delegate fsGroup correctly:

I0910 10:27:28.655774       1 nodeserver.go:209] NodeStageVolume: targetPath(/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/4a32186b174aa41c2369dd747134fc00cb1514550a5e6e37f860ba407eab6120/globalmount) volumeID(samba-server.samba-server.svc.cluster.local/share#pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a##) context(map[csi.storage.k8s.io/pv/name:pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a csi.storage.k8s.io/pvc/name:claim csi.storage.k8s.io/pvc/namespace:default source://samba-server.samba-server.svc.cluster.local/share storage.kubernetes.io/csiProvisionerIdentity:1725963559773-1299-smb.csi.k8s.io subdir:pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a]) mountflags([file_mode=0777 mfsymlinks cache=strict noserverino]) mountOptions([file_mode=0777 mfsymlinks cache=strict noserverino gid=1002])

adding gid=1002 to mount options. And gid=1002 is visible inside the Pod:

$ oc exec -it fedora -- bash
bash-5.2$ grep claim /proc/mounts
//samba-server.samba-server.svc.cluster.local/share/pvc-62d25ec6-c20e-45f8-815e-749c7df44b9a /mnt/claim cifs rw,relatime,vers=3.1.1,cache=strict,username=sambauser,uid=0,noforceuid,gid=1002,noforcegid,addr=172.30.35.246,file_mode=0777,dir_mode=0755,soft,nounix,mapposix,mfsymlinks,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1 0 0

However, the mount point is not writeable for the group:

bash-5.2$ ls -ld /mnt/claim 
drwxr-xr-x. 2 root 1002 0 Sep 10 13:35 /mnt/claim

I cannot see any workaround or solution to overcome this issue. Do you have any?

mpatlasov added a commit to mpatlasov/csi-driver-smb that referenced this issue Sep 13, 2024
@mpatlasov
fix: mount options for fsGroup delegation must ensure RWX for the group
b5b4ae3 
If user sets `fsGroup: <gid>` in Pod's spec.securityContext, kubelet delegate fsGroup to CSI Driver, and NodeStageVolume() adds `gid=<gid>` to mount options. This might be not enough to make volume writable for the user:
```
$ kubectl exec fedora -- ls -ld /mnt/claim
drwxr-xr-x. 2 root 1002 0 Sep 13 12:04 /mnt/claim
$ kubectl exec fedora -- touch /mnt/claim/FILE
touch: cannot touch '/mnt/claim/FILE': Permission denied
```
See kubernetes-csi#835
@mpatlasov mpatlasov linked a pull request Sep 13, 2024 that will close this issue
Open
@mpatlasov
Copy link
Contributor Author

@andyzhangx I'd appreciate your feedback on #848 . It fixes the issue by setting file_mode and dir_mode mount options. If this approach is tolerable, I'll add unit-tests to the PR. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: mount options for fsGroup delegation must ensure RWX for the group mpatlasov/csi-driver-smb
2 participants
@andyzhangx @mpatlasov