Consider auto-configuring Refinery memory consumption tunables by default

[bixu]

Is your feature request related to a problem? Please describe.
I've seen the tuning for MaxAlloc and CacheCapacity cause some confusion for operators of Refinery in Kubernetes.

Since sane default configs are documented here, why not leverage Helm to configure these?

Describe the solution you'd like
Since Refinery pods should get Guaranteed QoS in K8s anyway (requests and limits on RAM should be matching values), we could use those RAM allocation values to calculate the default values for both MaxAlloc and CacheCapacity. I think we'd want these to be overridable, however, via something like a custom values.yaml file.

Describe alternatives you've considered
I'm considering opening a PR on this myself, but need to negotiate for the time to do so from $job.

[seh]

You can push the memory limit value down into the Refinery container via the Kubernetes Downward API.

For now, we generate the Refinery configuration from the same program that generates a kustomize patch to set the memory request and limit, so we perform the recommended 80% calculation when writing the configuration file, but it would be nice to have Refinery derive these values from a memory limit supplied via environment variable.

[MikeGoldsmith]

Thanks for the suggestion. We'll add this to our backlog for prioritisation, or if you feel inclined we're always open to PRs.

[TylerHelmuth]

With Refinery 2.0's introduction of AvailableMemory and MaxMemoryPercentage the chart is now setting some memory defaults using the resource limits:

Collection:
    # AvailableMemory is the amount of system memory available to the Refinery process.
    AvailableMemory: '{{ .Values.resources.limits.memory }}'
    MaxMemoryPercentage: 75