Support envoy pod template with basic fields

[r0bj]

Please describe the problem you have
[A clear, concise, description of the problem you are facing. What is the problem that feature X would solve for you?]

It would be useful to be able to specify envoy pod template with basic fields like:

• labels
• annotations
• securityContext
• resources
  etc.

In some environments/clusters there are specific requirements for pod labels (e.g. mandatory label team, owner, contact, etc.). Other use case is for specific annotation which can be used to define pod log destination or other parameters.

[danehans]

@r0bj in my experience template-based approaches become very difficult to manage. Instead of exposing a "pod template" through the API, I would like to address each of the above bullets in a specific manner. Using labels as an example, we can have the operator append labels defined in the Contour to the operands:

apiVersion: operator.projectcontour.io/v1alpha1
kind: Contour
metadata:
  name: contour-sample
  namespace: projectcontour
  labels:
    environment: production
    app: foo
spec:
...

# Results in operands having the labels:
apiVersion: apps/v1
kind: DeamonSet
metadata:
  name: envoy
  namespace: projectcontour
  labels:
    environment: production
    app: foo
 ...

A similar approach can be taken for custom annotations.

We already have an issue for managing SCCs: #112.

What do you think about these approaches?

[r0bj]

Sure, it would be less flexible but it would work. Partly my inspiration was prometheus-operator where you can define podMetadata and also separate fields for: affinity, tolerations, securityContext, resources, etc. Similarly cert-manager and its ClusterIssuer: you can configure podTemplate with labels, annotations, nodeSelector, etc.

[danehans]

xref #373 for tolerations/nodeSelector support of Envoy pods.