Envoy container resource limits

[xpepermint]

I think it's reasonable to allow resources limits for the envoy container, especially when Contour is deployed to multiple namespaces. Do you agree? Is there such a configuration option already available? There should be a "scaling" section (as best practice guide) somewhere on the website to make the scaling/limits clear.

[danehans]

@xpepermint I would like to see resource limits added to the Contour example manifests before adding them to the operator. According to this reference, it's been a while since a resource baseline was established. @youngnick @stevesloka should a projectcontour/contour issue be created to update the guide and/or set resource limits for the example manifests?

[danehans]

xref: projectcontour/contour#409

[danehans]

@xpepermint unlike with user workloads, setting limits for cluster infrastructure components, i.e. Envoy, is problematic for several reasons:

• Components cannot anticipate how they scale in usage in all environments, so setting one-size-fits-all limits is not possible.
• Setting static limits prevents administrators from responding to changes in their clusters’ needs, such as by resizing control plane nodes to provide more resources.
• We do not want cluster components to be restarted based on their resource consumption (for example, being killed due to an out-of-memory condition). We need to detect and handle those cases more gracefully, without degrading cluster performance.

Therefore, it's my opinion that cluster components SHOULD NOT be configured with resource limits. However, the operator may be able to support this use case. Here are a few options:

• Remove resource limits from being managed by the operator. This will allow users to manually set the limits after the operator creates the Envoy Daemonset.
• Expose container resource configuration through the Contour API. I am concerned that this opens the door for users to break things by OOM'ing the Envoy containers.

Thoughts?

cc: @stevesloka @skriss @sunjayBhatia @Miciah

[xpepermint]

Got it @danehans, makes sense.