cost calculation wrong if a mix of spot and demand instances is used

[tfriedel]

Version

master (commit: 0657881)

Description

I have set a node group to include at least one on-demand instance and additional spot instances.
During cost calculation cortex assumes I have only spot instances and thus reports lower costs.

Configuration

relevant part in cluster.yaml:

  - name: gpu
    instance_type: g4dn.xlarge
    min_instances: 0
    max_instances: 5
    priority: 1
    instance_volume_size: 50
    instance_volume_type: gp3
    spot: True
    spot_config:
      on_demand_base_capacity: 1
      on_demand_percentage_above_base_capacity: 0
      instance_distribution: [g4dn.xlarge, g4dn.2xlarge]

Steps to reproduce

Setup cluster to to have a node group configured with "spot: True" and "on_demand_base_capacity: 1".
Deploy an api in this cluster so that at least one pod is deployed. Check price for this instance.

Expected behavior

The price for running one node should be the price listed for an on-demand instance.

Actual behavior

The price is a spot-instance price.

Suggested solution

Presumably during cost calculation only the flag "spot: True" is used. Instead the cost per instance needs to be queried from AWS.