Dynamic allocation

[cmelone]

We would like to use historical resource utilization data to predict future usage, and assign container memory and CPU requests within Kubernetes -- to optimize utilization.

To ensure we don't have negative impacts on the CI, we are starting with requests.

Because we are simply creating resource requests, it will still be possible for jobs to use more CPU/RAM than we expect. The next phase of experimentation will focus on instituting limits.

The ability to predict usage and assign requests has been implemented in the gantry web api, but there are a few steps needed to complete this goal.

• Merge CI scriptable config into spack
• Perform tests to ensure the system will function as expected
• Enable gantry in spack
• Monitor results to see if resources are indeed being optimized and fix any issues
• Implement assigning resource limits after designing an accurate prediction algorithm