Optimizing scheduling of nodes and pods

[cmelone]

Once dynamic allocation, retries, cost measurement, and fuzzing is implemented in gantry, we would like to turn the focus towards the scheduling of instances and pods.

Investigate queuing build jobs instead of the current k8s/karpenter approach where nodes are packed based on pod resource requests and where pods are not expected to wait long for assignment to an instance.

We should look into batching jobs into nodes with similar predicted build times using a custom packing algorithm. This would require some sort of pre-analysis when a pod is about to be scheduled.

Additionally, we should see if there are EC2 instances that are better for building specific packages. Should we consider removing any instance types from the karpenter pool?