[Question] Creating encrypted backups from encrypted ZFS pools

[LukasKnuth]

What steps did you take and what happened:

I'm using OpenEBS ZFS-localPV

1. Added a new zpool with sudo zpool create -o ashift=12 -o feature@encryption=enabled -O encryption=on -O keylocation=file:///root/zfs-encrypt.key -O keyformat=raw encrypted-pool `sudo losetup -f /tmp/zfs-encrypted.img --show`
2. Created a new StorageClass to create PVCs for this pool
3. Setup a new PVC from the storage class and wrote some plain data into it
4. Ran a Velero backup velero backup create encrypted-test --snapshot-volumes --include-namespaces=apps --volume-snapshot-locations=default --storage-location=default
5. The backup completed successfully and the data is found on my S3 storage
6. Downloaded the zfs-pvc-0828badb-1386-4869-a475-00f9795d262d-encrypted-test file from the S3 bucket (UUID matches my PVC on the cluster)
7. Ran strings zfs-pvc-0828badb-1386-4869-a475-00f9795d262d-encrypted-test | grep find_me and found the contents of the file on the encrytped PVC

What did you expect to happen:

The strings command doesn't print the contents of the file backed up from the encrypted pool.

The output of the following commands will help us better understand what's going on:

$ kubectl get storageclass/openebs-zfs-encrypted -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfs-encrypted
  uid: 6a79fea8-7bcc-4ea0-a609-162b0489a25c
parameters:
  dedup: "off"
  fstype: zfs
  poolname: encrypted-pool
provisioner: zfs.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

$ zfs get -p encryption,keystatus encrypted-pool
NAME            PROPERTY    VALUE        SOURCE
encrypted-pool  encryption  aes-256-gcm  -
encrypted-pool  keystatus   available    -

$ zfs get -p encryption,keystatus encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test
NAME                                                                    PROPERTY    VALUE        SOURCE
encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test  encryption  aes-256-gcm  -
encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test  keystatus   available    -

$ kubectl -n apps get pvc/encrypted-storage
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            AGE
encrypted-storage   Bound    pvc-0828badb-1386-4869-a475-00f9795d262d   1Gi        RWO            openebs-zfs-encrypted   53m

Anything else you would like to add:

Since there is no specific documentation on this subject in either this nor the drivers repository, I'm not sure if I might just have misunderstood or misconfigured something.

What I'm trying to do is have both encrypted ZFS filesystems backing my PVCs on the actual disk AND have the backup be encrypted in the cloud as well! Meaning it's not necessarily possible to restore a backup (fully) without the encryption key from the host (specified when creating the zpool) and the data on my PVCs is encrypted at rest.

Environment:

• Velero version (use velero version): 1.9.0
• Velero features (use velero client config get features): NOT SET
• Velero-plugin version: 3.3.0
• OpenEBS version: 2.1.0
• Kubernetes version (use kubectl version): v1.23.6
• Kubernetes installer & version: v1.24.3+k3s1
• Cloud provider or hardware configuration: Raspberry Pi 4
• OS (e.g. from /etc/os-release): Ubuntu 20