Prepare for volsync block device test (RamenDR#1559)

* Allow unprivileged pods to access block devices

We need this for testing volsync with block devices with minikube
clusters.

For minikube clusters this is done via the environment file which is
nicer, but it requires configuring containerd after the cluster has
started, which can cause failures in addons scripts.

We need to upstream this change to minikube later.

Signed-off-by: Nir Soffer <[email protected]>

* Log replication source status in yaml

It is easier to read and works better for viewing the replication logs
in the status.

Signed-off-by: Nir Soffer <[email protected]>

* Improve volsync test teardown

- Delete the replication source before unexporting the volsync service
  using it.
- Log every every teardown step to make debugging easier.

Signed-off-by: Nir Soffer <[email protected]>

* Fail if deleting destination namespace get stuck

Replace delete used for waiting for waiting on the deleted namespace
with a timeout. If deletion get stuck, the test will fail instead of
blocking forever, breaking stress test.

When delete get suck we can inspect the resources in the test gather
directory:

    % tree out/013.gather/dr2/namespaces/busybox
    out/013.gather/dr2/namespaces/busybox
    └── snapshot.storage.k8s.io
        └── volumesnapshots
            └── volsync-busybox-dst-dst-20240914203905.yaml

    % cat out/013.gather/dr2/namespaces/busybox/snapshot.storage.k8s.io/volumesnapshots/volsync-busybox-dst-dst-20240914203905.yaml
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    metadata:
      creationTimestamp: "2024-09-14T20:39:05Z"
      deletionGracePeriodSeconds: 0
      deletionTimestamp: "2024-09-14T20:39:05Z"
      finalizers:
      - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
      generation: 2

This looks like an external-snapshotter bug since volsync deleted the
snapshot and removed its finalizers.

Signed-off-by: Nir Soffer <[email protected]>

---------

Signed-off-by: Nir Soffer <[email protected]>

test/addons/volsync/test

@@ -7,6 +7,8 @@ import json
 import os
 import sys
 
+import yaml
+
 import drenv
 from drenv import kubectl
 from drenv import subctl
@@ -130,32 +132,43 @@ def run_replication(cluster):
     )
     status = json.loads(out)
     print("Replication status:")
-    print(json.dumps(status, indent=2))
+    print(yaml.dump(status))
 
 
 def teardown(cluster1, cluster2):
     """
     Remove deployments from both clusters. This also deletes additonal
     resources created in the same namespace.
     """
-    print("Cleaning up clusters")
+    print(f"Delete replication source in cluster '{cluster1}'")
+    kubectl.delete("--filename", "source/replication-src.yaml", context=cluster1)
+
+    print(f"Unexporting volsync service in cluster '{cluster2}'")
     subctl.unexport("service", VOLSYNC_SERVICE, cluster2, namespace=NAMESPACE)
+
+    print(f"Delete source in cluster '{cluster1}'")
     kubectl.delete(
         "--kustomize",
         "source",
         "--ignore-not-found",
         "--wait=false",
         context=cluster1,
     )
+
+    print(f"Delete destination in cluster '{cluster2}'")
     kubectl.delete(
         "--kustomize",
         "destination",
         "--ignore-not-found",
         "--wait=false",
         context=cluster2,
     )
-    kubectl.delete("--kustomize", "source", "--ignore-not-found", context=cluster1)
-    kubectl.delete("--kustomize", "destination", "--ignore-not-found", context=cluster2)
+
+    print(f"Waiting until namespace '{NAMESPACE}' is deleted in cluster '{cluster1}'")
+    kubectl.wait("ns", NAMESPACE, "--for=delete", "--timeout=120s", context=cluster1)
+
+    print(f"Waiting until namespace '{NAMESPACE}' is deleted in cluster '{cluster2}'")
+    kubectl.wait("ns", NAMESPACE, "--for=delete", "--timeout=120s", context=cluster2)
 
 
 if len(sys.argv) != 3:

test/envs/regional-dr.yaml

@@ -16,6 +16,10 @@ templates:
   - name: "dr-cluster"
     driver: "$vm"
     container_runtime: containerd
+    containerd:
+      plugins:
+        io.containerd.grpc.v1.cri:
+          device_ownership_from_security_context: true
     network: "$network"
     cpus: 4
     memory: "6g"