OCPNODE-2205: Lazy image pull support

[harche]

xref : https://issues.redhat.com/browse/OCPNODE-2205

[openshift-ci-robot]

@harche: This pull request references OCPNODE-2205 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

xref : https://issues.redhat.com/browse/OCPNODE-2205

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[harche]

/cc @sairameshv

[giuseppe]

our tools do not support eStargz. We are not able to produce this format.

We can use the zstd:chunked format.

Same comment applies in other places in this patch

[harche]

Updated. Thanks.

[ktock]

Stargz store performs lazy pulling if the image is eStargz or zstd:chunked and CRI-O performs lazy pulling for them when stargz store enabled. So I think the formats don't need to be limited to zstd:chunked and eStargz can still be listed as one of the lazy-pullable formats.

[harche]

@ktock to begin with only zstd:chunked format be supported by OCP support. i.e. we will not technically prevent anyone from using any other format, but the official support will only be extending, at least in the beginning, to zstd:chunked format.

[rphillips]

I think we should have the type in addition to the pool name. Just in case we want to change the type of the lazy image pull later.

[kannon92]

We would need this for KEP-4191 also. Since we are going to modify imagestore.

[kannon92]

It would be worth exploring what types of images would not work with stargz. And how we can deduce that?

ie Can we understand why stargz is not working for this images and what kind of logs can we look at to determine that for support/operational?

[harche]

That snapshotter supports zstd:chunked and eStargz format. OCP will support zstd:chunked format only. You can enable debugging in /etc/container/storage.conf to get more verbose output of the image handling.

[kannon92]

That isn't really answering my question.

Potential Benefits for Large Images: If the container image is shared between various containers which > utilize only part of the image during their execution, Lazy image pulling speeds up such container startup by > downloading only the relavent bits from those images.

this makes it sound like the benefits of lazy pulling is not guaranteed. So i wanted to know how we could know if it wasn't working or working. From a support and a user perspective.

[kannon92]

Will it always work if someone uses zstd:chunked for their containers. And in all cases, that is all that is needed?

Are there cases where all the bits would need to be downloaded for a container to start?

[harche]

If the Stargz service plugin is not running on the host, then the images will not get lazy pulled. i.e. the container would start times will be like regular container images.

If that plugin is running and the users suspect any defect in the plugin they can look at the unit logs for details.

You can use image with zstd:chucked format, but without that plugin the lazy part of image pulling won't work, so you would have container start up times comparable to the regular container images.

[kannon92]

I'm mostly asking because the language you said makes it sound like this isn't guaranteed.

I think if stargz service plugin is enabled, and images will get lazy pulled.

I would maybe just drop the "Potentially".

But adding some logs on debugging this would be useful for docs/feature enablement.

[mtrmac]

Basically lazy pulling is trading throughput for latency — startup might be faster, but all operations are a bit slower due to the indirection required for the “is the file available already, or do I need to download it” checks.

So, the answer to “what types of images don’t benefit” from lazy pulling is “all well-designed ones” (when throughput matters): if all files in the image are actually necessary, and consumed over the lifetime of the application, then it is faster to pull and to use the image using an ordinary pull, rather than the FUSE slowly-download-files-on-demand lazy approach.

Lazy pulling should be beneficial only if many files in the container are not actually necessary for processing the typical request (and if the files that are necessary are so few that the per-file request latency is not overwhelming). I’m sure that happens frequently in practice, but it is quite likely an indication of something that could be fixed in the image.

---

(There are other benefits to zstd:chunked, like possibly sharing files across container images / layers, but those don’t depend on lazy pulling.)

[harche]

Basically lazy pulling is trading throughput for latency

stargz-snapshotter will continue to download rest of the image in the background.

[mtrmac]

The FUSE overhead is paid during every single filesystem operation during the container runtime, isn’t it? For a long-running container which is not running purely out of memory (e.g. a single statically linked binary) it will eventually overwhelm any possible savings.

[harche]

/test markdownlint

[cgwalters]

(why?)

[harche]

For the dev preview phase we thought it would be better to limit the usage to the worker nodes because we still have an important pending issue of downloaded chucks not being verified in backing c/storage library. @giuseppe, @ktock and @mtrmac are working on addressing that - containers/storage#795 (comment)

We will be blocking on that before graduating this to TechPreview

[mtrmac]

FWIW broken layer integrity enforcement is a security blocker.

I don’t know that it’s appropriate to ship to customers in any channel; at the very least it would require a well-advertised caveat that customers certainly see before actively opting in.

[mtrmac]

(Also, at the very least I am not actively working on that, I don’t know about the others listed.

As a blocker for this enhancement it probably should be tracked in more salient place than a closed upstream PR.)

[cgwalters]

I am curious how "battle tested" this is. I guess we'll find out in testing, but the amount of new moving parts in code here has me left with an instinct that this will require a maintainer.

[harche]

I am curious how "battle tested" this is. I guess we'll find out in testing

Initial tests in OCP are promising. We are also proposing to add Openshift e2e job around lazy image pulling in this EP.

[mtrmac]

I agree, the plugin will certainly require a knowledgeable maintainer able to deal with customer support escalations. Even if there were no bugs in it (which I doubt), expertise would still be required to allow post-mortem analysis of broken systems.

That’s a non-trivial ongoing cost.

[cgwalters]

I know what you're trying to say here but it sounds like it's implying zstd:chunked is separate from OCI; as you know it's just an alternative compression scheme with metadata, but readers may not. How about

Replacement of the standard OCI image format; zstd:chunked is a variant of zstd compressed layers in OCI, which is supported by podman and moby

[harche]

Sounds great, thanks. Updated.

[cgwalters]

Since they can only specify one thing, why would this be mentioned in the config?

[harche]

Currently we have eStargz and zstd:chunked formats supported by the stargz-snapshotter. We are going to support only zstd:chucked format to begin with. But in future, we may support other snapshotters with their own image formats e.g. https://github.com/awslabs/soci-snapshotter

So we are just trying to make sure in future we should be able to support more snapshotters and formats (if required).

/cc @rphillips

[cgwalters]

Why? That's bizarre.

[harche]

Thanks, dropped it.

[cgwalters]

This means very little and I doubt it's necessary https://systemd.io/NETWORK_ONLINE/

[harche]

Thanks, dropped it.

[cgwalters]

For us actually, WantedBy=crio.service or kubelet-dependencies.target.

[harche]

Thanks, updated to crio.service

[cgwalters]

Right so here's where I think what we're doing in splitting binaries in CoreOS and configuration in MCO makes little sense.

I think if this is packaged as an RPM then this systemd unit should live in that RPM and not in the MCO.

And most importantly, it's ExecStart=/usr/bin/stargz-store.

(/usr/local is machine local state on CoreOS)

[harche]

Updated ExecStart=/usr/bin/stargz-store, thanks.

If we ship the systemd unit with the RPM, is there a way to enable/disable the service using MCO? We would want this service only if user explicitly wants to use lazy image pulling. Otherwise, there is no reason to have that service running by default.

[cgwalters]

See also https://www.phoronix.com/news/Linux-6.9-FUSE-Passthrough

[harche]

Thanks for that pointer, added a remark in that section about it at the end.

[cgwalters]

Yes. But...I think a notable callout here is going to need to be the docs work to describe how to do that.

In general it should be highlighted here more obviously that because zstd:chunked isn't the default, in reality customers will need to enable it in their image builds for their own images, and if they want the benefit for our images, they'll need to re-compress (which is probably not very viable).

[harche]

Good idea, I will add a documentation related goal so that we won't miss it.

[mtrmac]

and if they want the benefit for our images, they'll need to re-compress (which is probably not very viable).

In particular recompressing the OpenShift product images is, AFAIK, not viable in production workloads, because the release image payload, and the image integrity design, is built around the release image containing a full list of image digests of the OpenShift images.

[mtrmac]

Similarly any signed image. And I’d expect operators / Helm charts to fairly frequently refer to images by digest.

So… this will most likely primarily apply to customer-internal images only.

[rphillips]

format: zstd:chunked

[harche]

That will support only one, but in future if you want to support one or more then the list would be helpful.

format:
    - "zstd:chucked"
    - "ystd:hunked"
    - "xstd:junked" 

[harche]

https://github.com/openshift/enhancements/pull/1600/files#r1550258088

[ktock]

Stargz store also supports eStargz and CRI-O performs lazy pulling for both of zstd:chunked and eStargz so can we list that format here as well?

[harche]

@ktock Thanks for your reply. This enhancement proposal is aimed at introducing and supporting the new feature in Openshift. While it is true that the stargz snapshotter supports both zstd:chunked and eStargz formats, from the Openshift's supportability point of view we have to evaluate overall ecosystem around any feature and how best we can support our customers. At this point in time, Redhat's main container image building tools like podman and buildah focus on supporting zstd:chunked image format.

But having said that, we would definitely officially support more formats if overall container tools from Redhat support those formats as well. In fact, this is the reason we have added format field in the API with this enhancement proposal.

[ktock]

Thanks for the clarification!

This enhancement proposal is aimed at introducing and supporting the new feature in Openshift.

👍

While it is true that the stargz snapshotter supports both zstd:chunked and eStargz formats, from the Openshift's supportability point of view we have to evaluate overall ecosystem around any feature and how best we can support our customers. At this point in time, Redhat's main container image building tools like podman and buildah focus on supporting zstd:chunked image format.
But having said that, we would definitely officially support more formats if overall container tools from Redhat support those formats as well. In fact, this is the reason we have added format field in the API with this enhancement proposal.

So the runtime will have a logic to decide whether it enables lazy pulling on the pulling image, right? I have a question in the proposal, related to the implementation of that logic: PTAL #1600 (comment)

[harche]

We have a operator in Openshift called Machine Config Operator (MCO). We use that operator to lay down node specific configurations (e.g. config files in /etc). Our plan is to use that operator to lay down required configuration for the c/storage library used by crio to enable this feature. In fact, that config file for c/storage is already placed by MCO already, we just need to update it with the lazy image related configuration (if user decides to use this feature).

[ktock]

How is this value consumed by the runtime? i.e. How can the runtime use this value for deciding whether it enable lazy pulling on the pulling image? Maybe this should be a mediatype + manifest annotation?

[mtrmac]

Right now, none of this exists at a generic level: the on-demand layer plugin is provided with an image reference and a layer digest, and that’s it. So if this had to be implemented today, that plugin would need to re-parse the image manifest, find the layer, and impose its own conditions on the layer’s metadata.

If we need to add a generic feature, that will need to be co-designed along with this enhancement.

[harche]

How is this value consumed by the runtime? i.e. How can the runtime use this value for deciding whether it enable lazy pulling on the pulling image? Maybe this should be a mediatype + manifest annotation?

@ktock As of now, this values isn't directly used by the runtime. We are using this value to in MCO to prepare the configuration file for the c/storage library.

[mtrmac]

c/storage AFAIK has no option to restrict ALS lookups by compression format right now.

[ktock]

I think we can introduce a field additionallayerstore_target_mediatype in storage.conf. This is layer mediatypes to enable to lookup of Additional Layer Store.

containerRuntimeConfig.lazyImagePull.format should also be a list of mediatype. Or, MCO can define aliases for them; e.g. "zstd:chunked" means application/vnd.oci.image.layer.v1.tar+zstd.

[mtrmac]

My first instinct, from the runtime and c/storage perspective is to ask “why do we need that option at all?”.

I could, maybe, imagine that users might want to enable/disable lazy pulls based on the application (= on the registry namespace, on the K8S namespace, or as a Pod option). (To be clear, I don’t think we need that for a first version at all.)

But why would users want to make a choice based on the format? Isn’t that ~transparent to the running systems, other than performance?

---

It might be necessary to have an option that governs “what format backends (snapshotters?) should be installed on nodes” — maybe not for the first version, but it does make sense to ensure that we can add that option later.

But that option should govern the behavior of MCO, c/storage doesn’t need to know. storage.conf only contains an additionallayerstore option with a list of mount points; how those mount points get set up and what are the backing FUSE servers is not a storage.conf concern AFAICS.

[harche]

I agree, we can drop this option.

[ktock]

Thanks for the review. Removed this option from the proposal.

[mtrmac]

I don’t see any mechanism for containers to be downloaded again.

(Cc: @giuseppe to keep me honest about the c/storage aspects.)

Once an Image object exists in c/storage, it doesn’t go automatically away when a backend layer disappears; the image is just broken. Once a layer record exists in c/storage metadata, it doesn’t go automatically away when the layer store changes and the layer no longer exists; the layer is just broken (and might even “infect” future pulls that share this layer, because the metadata lookup will indicate that the layer already exists locally).

Unless the downgrade triggers a complete wipe? I don’t know whether it does, and the text above is not worded in a way consistent with that.

(Also, zstd in a non-chunked variant has been supported for a long time; and layers are locally stored uncompressed, so the compression format doesn’t matter all that much. But that’s all mostly irrelevant.)

Even in the absence of stargz-snapshotter, those container will start but without any performance gains.

Absolutely not; the snapshotted layers are stored in different paths from the point of view of the graph driver (and I very much hope that the backend plugins are not trying to store anything in the graph driver-managed locations).

See above about “once a layer record exists”…

This downgrade either needs to explicitly trigger a wipe (assuming that’s possible at all), or must be prevented.

---

And it’s not just downgrades: disabling the snapshotters on an already-running node would break the node for the same reasons.

[harche]

Yes, you are correct, if the images need to be cleaned up if the stargz store service is not enabled (or fails to start). I will update the proposal to explicitly call out that scenario. Thanks.

[mtrmac]

See the “downgrade” section; just disabling the option will break the node, AFAICS.

[mtrmac]

• Just to make sure OCPNODE-2205: Lazy image pull support #1600 (comment) is considered
• I don’t see any discussion of registry credentials. The lazy-pull layer store does not currently get any credentials from the actual pull operation in c/image, they must be provided to the plugin via other means
  • I see no discussion of that happening here at all
  • Assuming the capabilities as of today, that would either mean that per-image pull secrets linked from the Pod object would not be used at all, at best this would work for the cluster global pull secret; that’s possible but a fairly significant downside for any production workloads; or maybe this particular snapshotter seems to support somehow capturing credentials by interposing a CRI proxy (ingenious but eww, and I don’t know how that could work in multi-tenant situations).
  • Alternatively, some new feature to provide the credentials directly from the pull operation would have to be built. And because providing credentials via file paths passed to FUSE is out of the question, that would be a major rearchitecture of the API, AFAICS

[openshift-bot]

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

[ktock]

/remove-lifecycle stale

[openshift-ci-robot]

@harche: This pull request references OCPNODE-2205 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

xref : https://issues.redhat.com/browse/OCPNODE-2205

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lmzuccarelli for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ktock]

@harche Thanks for reopening this and giving me access to this PR.

@mtrmac Thanks for the comment about downgrading/upgrading.

layer_integrity_state_file = "<location>" layer_integrity_state_marker = ""

Thanks for the suggestion. This sounds good to me. Updated proposal to include this.

[mtrmac]

Cc: @haircommander @saschagrunert

WDYT about this mechanism? Is it potentially valuable for other use cases / overengineered / insufficient?

[cgwalters]

I think this type of thing should really be shared with podman and other c/storage consumers.

What this is trying to achieve is IMO very similar to "feature flags" as used by Linux filesystems for example for many years, with explicit concept of e.g. "write-incompat" and "incompat" flags too.

For example, composefs is currently expressed as a use_composefs = "true" flag but when set up it's clearly on-disk incompatible with a c/storage consumer that predates support for that (or that doesn't have the flag enabled). Yet, we don't really (AFAIK) express that state anywhere except implicitly in the filesystem layout for image that happened to be pulled with composefs.

One interesting tension I see here actually is the split between "config for storage" and "storage" - and again if you look at how Linux filesystems (and for that matter sqlite) do it is that these "feature flags" are embedded in the storage itself. It'd make more sense to me to store these things in /var/lib/containers/storage/meta.json or whatever instead of in a config file.

[haircommander]

yeah this almost seems like something that should happen in a separate service

[mtrmac]

I think there are two elements to this:

• Did the configuration change in unexpected ways? That could be stored in c/storage, although the c/storage storage is not packaged into a neat atomic bundle that makes it trivial (the config file[1] points at the “root” store location [where meta.json could live], and possible other stores, and the lazy pull mount point; the config could change the “root” store location, or to switch the primary “root” with a secondary root).
• What to do about a detected configuration change? The policy “it is OK to just delete existing images” is specific to K8s or CRI-O, or possibly an even higher-level decision; it’s not something c/storage can, on its own, decide to do.

As an analogy, Podman has some kind of “essential config file entries” record in its primary database, and that does include (some of?) the c/storage config settings. I didn’t look into how that works in detail.

[1] and there is a RFE to allow drop-in config file stubs, so even the config file itself is not a natural location for a compatibility marker.

---

The reason I was thinking this should be a CRI-O feature is:

• It needs to run before CRI-O starts really responding to requests
• It needs to trigger CRI-O-visible changes, like deleting images (and containers?)
• It needs to use exactly the same storage configuration as CRI-O
• It needs to be ~stateless/idempotent, not a single-shot “wipe now”, so that MCO can ensure the wipe happens even if the node goes down during the first boot after a config change.

Maybe it can be a separate systemd service running before CRI-O; I don’t know whether that would make things any easier.

[cgwalters]

(the config file[1] points at the “root” store location

...

As an analogy, Podman has some kind of “essential config file entries” record in its primary database, and that does include (some of?) the c/storage config settings. I didn’t look into how that works in detail.

Encoding the file paths in the storage itself is a broken concept and did break in practice, see containers/podman#23447

But that's distinct from the other config options. I think it could very much make sense to support e.g. podman image config set composefs=true which would end up looking up the storage root path from the global config, and then flipping a flag on there that lives in that storage, as opposed to changing the global storage.

In the short term nothing stops crio from doing this itself with /var/lib/containers/storage/crio-state.json or whatever.

[saschagrunert]

Would that work together with the latest CRI-O wipe enhancements @kwilczynski ?

[ktock]

@mtrmac @haircommander

a separate service

If this should be a separated service, I think we can have a subcommand in CRI-O, described as the following:

• crio check-layer-state <layer_integrity_state_marker> <layer_integrity_state_file>

This will run as a separated service before CRI-O starts.
This does the same things as described in the original proposal.
Concretely:

• When it starts for the first time on the node, it creates a file (call it "state file") at the location specified by layer_integrity_state_file with the contents of layer_integrity_state_marker (call it "state marker").
• Everytime this command starts on the node, it checks that the specified layer_integrity_state_marker exactly matches the contents of the state file at layer_integrity_state_file. If it doesn't match, it starts repairing the storage.

We can also avoid adding layer_integrity_state_file and layer_integrity_state_marker to crio.conf.

WDYT about this mechanism?

[haircommander]

that works for me

[ktock]

Thanks for the review. Added the description about this subcommand to the proposal.

[ktock]

@mtrmac @haircommander I've posted an implementation of this command to cri-o/cri-o#8589 .

[haircommander]

it only does this on sudden reboot, and only as of 4.17 does it repair, before it just removes all the storage

[haircommander]

I'm not sure the repair sequence knows how to fix stargz images

[mtrmac]

Removing an inconsistent image would be sufficient. Removing all storage, I guess, acceptable?

[harche]

@ktock Can we direct user to a blog like this one, https://blog.cubieserver.de/2022/experimenting-with-estargz-image-pulling-on-openshift/, to use lazy image pull in Openshift instead of going through this route?

[ktock]

@harche Thanks for your comment, but I think a blog can't achieve features discussed here:

• Automation of deploying services & configurations (stargz-store service and c/storage config, etc.)
• Item 1 in the Goals list: Package upstream Stargz Store plugin for the installation on RHCOS

[harche]

@ktock packaging is only required if it has to be shipped with RHCOS officially. Nothing stops someone from downloading and installing a binary like that blog mentions.

[ktock]

@harche

Nothing stops someone from downloading and installing a binary like that blog mentions.

But now we need several binaries and configurations (stargz-store and auth helper) and this can be more complex. From the user's perspective, installing all of them correctly can be harder compared to the time that blog was written. Officially automated and packaged installation can help make the maintenance of nodes easier.

[harche]

@harche

Nothing stops someone from downloading and installing a binary like that blog mentions.

But now we need several binaries and configurations (stargz-store and auth helper) and this can be more complex. From the user's perspective, installing all of them correctly can be harder compared to the time that blog was written. Officially automated and packaged installation can help make the maintenance of nodes easier.

I agree with you. However, supporting enterprise customers becomes challenging if there aren't sufficient resources allocated to maintaining these feature (bugs, CVEs etc), especially since its priorities has shifted a bit.

[ktock]

I agree with you. However, supporting enterprise customers becomes challenging if there aren't sufficient resources allocated to maintaining these feature (bugs, CVEs etc), especially since its priorities has shifted a bit.

@harche Thanks for the comment. Of course, I'm willing to continually maintain lazy-pulling-related features and stargz-store from the community side! One possible way to make it easier would be to have a repo under github.com/containers org for receiving issues and for maintaining lazy-pulling-related resources (e.g. helpers).

[sairameshv]

I agree with you. However, supporting enterprise customers becomes challenging if there aren't sufficient resources allocated to maintaining these feature (bugs, CVEs etc), especially since its priorities has shifted a bit.

@harche Thanks for the comment. Of course, I'm willing to continually maintain lazy-pulling-related features and stargz-store from the community side! One possible way to make it easier would be to have a repo under github.com/containers org for receiving issues and for maintaining lazy-pulling-related resources (e.g. helpers).

Hey, FYI, there is already a repo/fork for the stargz-snapshotter here
We are in the process of packaging the required rpms from this repository.

[ktock]

@sairameshv Thanks for the info! Please tell me anything I can help from my side.

[harche]

@ktock @sairameshv are there any open comments on this enhancement proposal preventing it from merging?

[ktock]

I think currently there is no comment that prevents it from merging. (cc @mtrmac)

[mtrmac]

This LGTM as written, but I’m not an approver in this repo; and I can’t speak to support / production deployment implications.

[haircommander]

FYI openshift api doesn't anymore allow booleans because they're not extendable, you'll need to add an enum

[ktock]

Thanks for the review. Fixed to avoid using booleans.

[openshift-ci]

@harche: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[haircommander]

/lgtm

[ktock]

Thanks for your approval. Can we move this forward and merge this?

[harche]

Thanks for your approval. Can we move this forward and merge this?

cc @mrunalp for approval required to merge it.

[ktock]

@mrunalp Could you please take a look at this PR? 🙏

[harche]

@mrunalp Could you please take a look at this PR? 🙏

@ktock I think @mrunalp should be able to take a look at this when he is back next week.