Add skopeo-based adapter for working with OCI images

[kyleam]

This series adds an adapter that, like the existing docker-archive-based adapter, can be used to track a Docker image in a dataset. The disadvantage is that skopeo needs to be installed, while the main advantage is that the layers downloaded from docker:// sources can be linked to registry URLs.

This depends on support for registry URLs in datalad. Although the approach will likely change, there is a draft PR at datalad/datalad#5129.

Related:
#98
#135

Closes #106.

---

• Finalize last commit based on how linking changes land on datalad's side (NF: downloaders: Support downloading Docker registry blobs datalad#5129)
• Drop temporary base commit when in-flight PRs are merged
• Deal with conflicts when BF: docker adapter: Don't couple 'run -i' with stdin.isatty #142 lands
• Replace ensure_datalad_remote variant from this PR with the one added to core in 434e3f9a8a (ENH: customremotes: Add ensure_datalad_remote(), 2021-05-07), which will be included in the release following 0.14.4.

[codecov]

Codecov Report

Merging #136 (fc7b847) into master (0f41d68) will decrease coverage by 7.75%.
The diff coverage is 50.98%.

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
- Coverage   86.54%   78.79%   -7.76%     
==========================================
  Files          17       22       +5     
  Lines         929     1193     +264     
==========================================
+ Hits          804      940     +136     
- Misses        125      253     +128     

Impacted Files	Coverage Δ
datalad_container/adapters/docker.py	43.05% <28.57%> (+7.10%)	⬆️
datalad_container/adapters/tests/test_oci_more.py	30.30% <30.30%> (ø)
datalad_container/adapters/utils.py	30.76% <30.76%> (ø)
datalad_container/adapters/oci.py	42.67% <42.67%> (ø)
datalad_container/containers_add.py	89.85% <50.00%> (-3.16%)	⬇️
datalad_container/adapters/tests/test_oci.py	100.00% <100.00%> (ø)
datalad_container/tests/test_utils.py	100.00% <100.00%> (ø)
datalad_container/utils.py	100.00% <100.00%> (ø)
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f41d68...fc7b847. Read the comment docs.

[yarikoptic]

just a note (sorry if I am repeating what is already known etc): Installing "yet another tool" (which is not automagically installed from pypi) could challenge some users. I wondered if we could somehow simplify deployment of skopeo... but it seems to rely on a handful of libraries

$> ldd /usr/bin/skopeo
	linux-vdso.so.1 (0x00007fff1295c000)
	libgpgme.so.11 => /lib/x86_64-linux-gnu/libgpgme.so.11 (0x00007fd9aa6c6000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd9aa6a4000)
	libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007fd9aa637000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd9aa472000)
	libassuan.so.0 => /lib/x86_64-linux-gnu/libassuan.so.0 (0x00007fd9aa45c000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fd9aa436000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd9aa754000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fd9aa408000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fd9aa3e1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd9aa29d000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fd9aa20d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd9aa207000)

good news is that skopeo is available from debian and conda-forge : https://anaconda.org/conda-forge/skopeo so at least in those packaging/deployment scenarios we could streamline (add as a dependency or Recommends: in debian).

edit: just now mentioned -- no windows build on conda-forge, only osx and linux :-/ filed conda-forge/skopeo-feedstock#9

[kyleam]

I've pushed an update to avoid duplicating logic across the docker and skopeo adapters. The first eight commits of the series do that, with the first commit being a temporary commit with @adswa's Windows-related docker adapter fixes from gh-137 and gh-140.

range-diff

 -:  -------- >  1:  6074460a [drop] base for recent docker adapter PRs
 -:  -------- >  2:  22118280 MV: docker adapter: Move some bits to adapters.utils
 -:  -------- >  3:  01b1a202 RF: adapters.utils: Rename _list_images to get_docker_image_ids
 -:  -------- >  4:  c52a0c0c RF: adapters: Prefer subprocess.run over check_* methods
 -:  -------- >  5:  3072f6d3 MV: adapters: Move logging configuration to utils
 -:  -------- >  6:  faf5256c ENH: adapters.utils: Avoid double logging
 -:  -------- >  7:  59ed18f7 ENH: adapters.utils: Add helper for main() handling
 -:  -------- >  8:  94363806 ENH: adapters.utils: Display captured stderr on exit
 1:  ac3db32a !  9:  cdea6a52 NF: oci: Add adapter for working with OCI images
    @@ Commit message
         via podman in the future.

         Add an initial skopeo-based OCI adapter.  At this point, it has the
    -    same functionality as the docker adapter.  Structurally it's fairly
    -    similar to the docker adapter, but it's not clear that it's worth the
    -    effort to try to share any common parts, at least until the oci
    -    interface settles and we decide to keep the docker adapter around.
    +    same functionality as the docker adapter.

      ## datalad_container/adapters/oci.py (new) ##
     @@
    @@ datalad_container/adapters/oci.py (new)
     +
     +import json
     +import logging
    -+import os
     +from pathlib import Path
     +import re
     +import subprocess as sp
     +import sys
     +
    ++from datalad_container.adapters.utils import (
    ++    docker_run,
    ++    get_docker_image_ids,
    ++    log_and_exit,
    ++    setup_logger,
    ++)
    ++
     +lgr = logging.getLogger("datalad.container.adapters.oci")
     +
     +
    @@ datalad_container/adapters/oci.py (new)
     +    return info["config"]["digest"]
     +
     +
    -+def _get_docker_image_ids():
    -+    out = sp.run(
    -+        ["docker", "images", "--all", "--quiet", "--no-trunc"],
    -+        capture_output=True, universal_newlines=True, check=True)
    -+    return out.stdout.splitlines()
    -+
    -+
     +def load(path):
     +    """Load OCI image from `path`.
     +
    @@ datalad_container/adapters/oci.py (new)
     +    An image ID (str)
     +    """
     +    image_id = get_image_id(path)
    -+    if image_id not in _get_docker_image_ids():
    ++    if image_id not in get_docker_image_ids():
     +        lgr.debug("Loading %s", image_id)
     +        # The image is copied with a datalad-container/ prefix to reduce the
     +        # chance of collisions with existing names registered with the Docker
    @@ datalad_container/adapters/oci.py (new)
     +
     +def cli_run(namespace):
     +    image_id = load(namespace.path)
    -+    prefix = ["docker", "run",
    -+              # FIXME: The -v/-w settings are convenient for testing, but they
    -+              # should be configurable. Another option would be to not specify
    -+              # any arguments to docker-run and let everything be handled by
    -+              # the caller.
    -+              "-v", "{}:/tmp".format(os.getcwd()),
    -+              "-w", "/tmp",
    -+              # Make it possible for the output files to be added to the
    -+              # dataset without the user needing to manually adjust the
    -+              # permissions.
    -+              #
    -+              # FIXME: But if the container doesn't have these IDs, some
    -+              # commands may fail.
    -+              "-u", "{}:{}".format(os.getuid(), os.getgid()),
    -+              "--rm"]
    -+    if sys.stdin.isatty():
    -+        prefix.extend(["--interactive", "--tty=true"])
    -+    prefix.append(image_id)
    -+    cmd = prefix + namespace.cmd
    -+    lgr.debug("Running %r", cmd)
    -+    sp.run(cmd, check=True)
    ++    docker_run(image_id, namespace.cmd)
     +
     +
     +def main(args):
    @@ datalad_container/adapters/oci.py (new)
     +
     +    namespace = parser.parse_args(args[1:])
     +
    -+    lgr.setLevel(logging.DEBUG if namespace.verbose else logging.INFO)
    ++    setup_logger(lgr, logging.DEBUG if namespace.verbose else logging.INFO)
     +
     +    namespace.func(namespace)
     +
     +
     +if __name__ == "__main__":
    -+    try:
    ++    with log_and_exit(lgr):
     +        main(sys.argv)
    -+    except Exception as exc:
    -+        lgr.exception("Failed to execute %s", sys.argv)
    -+        if isinstance(exc, sp.CalledProcessError):
    -+            excode = exc.returncode
    -+            if exc.stderr:
    -+                sys.stderr.write(exc.stderr)
    -+        else:
    -+            excode = 1
    -+        sys.exit(excode)
 2:  62c3e97e = 10:  9013fdf6 DOC: oci: Add todo comment about image ID mismatch
 3:  5710e8f8 <  -:  -------- ENH: oci: Wire up logging handler when script called via path
 4:  1bbfac5d = 11:  27cd9417 DOC: oci: Add comment about alternative destinations
 5:  8cae9302 = 12:  02fa4922 ENH: oci: Silence 'skopeo copy' output when loading image
 6:  45a7d332 ! 13:  954b5d84 ENH: oci: Add utility for parsing Docker reference
    @@ datalad_container/adapters/oci.py
     +from collections import namedtuple
      import json
      import logging
    - import os
    + from pathlib import Path
     @@
      lgr = logging.getLogger("datalad.container.adapters.oci")

 7:  a51391b9 = 14:  47a2054e ENH: oci: Copy over tag when saving OCI directory
 8:  cb262015 = 15:  aa7653a3 ENH: oci: Add utilities for storing and reading annotation field
 9:  c398c789 = 16:  a04e6a5f ENH: oci: Record source for skopeo-copy in image's annotation
10:  fa6b5388 = 17:  d2f33fd4 ENH: oci: Register a more informative name with docker-daemon
11:  3d670b99 ! 18:  ee1a33af ENH: containers-add: Wire up OCI adapter
    @@ datalad_container/adapters/tests/test_oci_more.py (new)
     +    WitlessRunner,
     +)
     +from datalad_container.adapters import oci
    ++from datalad_container.adapters.utils import get_docker_image_ids
     +from datalad.tests.utils import (
     +    assert_in,
     +    integration,
    @@ datalad_container/adapters/tests/test_oci_more.py (new)
     +
     +    image_path = ds.repo.pathobj / ".datalad" / "environments" / "bb" / "image"
     +    image_id = oci.get_image_id(image_path)
    -+    existed = image_id in oci._get_docker_image_ids()
    ++    existed = image_id in get_docker_image_ids()
     +
     +    try:
     +        out = WitlessRunner(cwd=ds.path).run(
12:  a5a13621 = 19:  b4d73a17 MV: Add utils module with containers-add's _ensure_datalad_remote
13:  290743a9 ! 20:  c45cef66 ENH: containers-add: Try to link layers in OCI directory
    @@ datalad_container/adapters/tests/test_oci_more.py
     +    DATALAD_SPECIAL_REMOTES_UUIDS,
     +)
      from datalad_container.adapters import oci
    + from datalad_container.adapters.utils import get_docker_image_ids
      from datalad.tests.utils import (
          assert_in,
          integration,

[mih]

I am closing this draft after some years. It is evident that nobody is working on it, and #106 exists to remind that this effort was made.