Support mixed MIG+MPS dynamic partititioning

[Telemaco019]

Description

Currently, when enabling Dynamic GPU Partitioning on a node, it is possible to choose only between MIG or MPS by adding one of the following labels: nos.nebuly.com/gpu-partitioning: "mig" or nos.nebuly.com/gpu-partitioning: "mps".

It would be nice to have a third dynamic partitioning option that mixes MIG and MPS. This would be particularly useful for further partitioning MIG devices with MPS, as often the smallest available MIG device on a GPU is way larger than the resources required by the workloads.

For instance, the smallest MIG profile for NVIDIA-A100-SXM4-80GB is 1g.10gb, which provides 10GB of GPU memory. However, since many workloads require less than 10GB of GPU memory, this leads to inefficiencies.

Right now the alternative is to partition GPUs using MPS, which allows the creation of GPU slices of arbitrary size. However, MPS does not provide full workload isolation. Using MPS on top of MIG would enable finer-grained partitioning without compromising too much workload isolation, as only the workloads sharing the same MIG partition wouldn't be fully isolated.

Proposed solution

Add the possibility to label a node with nos.nebuly.com/gpu-partitioning: "mixed". For nodes with this label, nos should automatically use MPS for partitioning the smallest available MIG devices according to the requested resources.