AWS can return multiple values per instance-type per region
shrinandj opened this issue · 0 comments
shrinandj commented
We recently found that the minion-manager was not switching from on-demand to spot-instances because it saw that the spot-instance price was > than the on-demand price. The on-demand instance price was seen to be 0.000 :-).
Turns out that this was happening because the AWS on-demand pricing endpoint was returning multiple values for that instance-type for the same region. The current mechanism for gathering the ondemand instance price will simply take the price of the last entry for that instance type. But it seems that that price could be 0.00.
This is what we got in one case:
| {'SKU': '2N2QH6UEJZ5GUPT8' | 'OfferingClass': '' | 'Group': '' | 'Instance Capacity - xlarge': '' | 'Instance Capacity - 16xlarge': '' | 'PricePerUnit': '0.0464000000' | 'PriceDescription': '$0.0464 per On Demand Linux t2.medium Instance Hour' | 'Storage': 'EBS only' | 'Pre Installed S/W': 'NA' | 'Instance': '' | 'Normalization Size Factor': '2' | 'Location': 'US East (Ohio)' | 'Memory': '4 GiB' | 'Physical Processor': 'Intel Xeon Family' | 'operation': 'RunInstances' | 'Dedicated EBS Throughput': '' | 'Instance Capacity - 10xlarge': '' | 'Instance Capacity - 4xlarge': '' | 'To Location': '' | 'From Location': '' | 'Operating System': 'Linux' | 'Product Family': 'Compute Instance' | 'GPU': '' | 'Intel Turbo Available': '' | 'Intel AVX Available': '' | 'Max IOPS Burst Performance': '' | 'Instance Capacity - 32xlarge': '' | 'ECU': 'Variable' | 'Tenancy': 'Shared' | 'Instance Capacity - 18xlarge': '' | 'OfferTermCode': 'JRTCKXETXF' | 'Instance Capacity - 9xlarge': '' | 'Instance Capacity - 8xlarge': '' | 'Processor Architecture': '32-bit or 64-bit' | 'EBS Optimized': '' | 'Group Description': '' | 'Provisioned': '' | 'Location Type': 'AWS Region' | 'EffectiveDate': '2018-10-01' | 'License Model': 'No License required' | 'vCPU': '2' | 'TermType': 'OnDemand' | 'instanceSKU': '' | 'PurchaseOption': '' | 'Instance Type': 't2.medium' | 'Instance Capacity - 2xlarge': '' | 'LeaseContractLength': '' | 'Instance Capacity - large': '' | 'StartingRange': '0' | 'Max IOPS/volume': '' | 'Max throughput/volume': '' | 'To Location Type': '' | 'Processor Features': 'Intel AVX; Intel Turbo' | 'Intel AVX2 Available': '' | 'GPU Memory': '' | 'serviceName': 'Amazon Elastic Compute Cloud' | 'Network Performance': 'Low to Moderate' | 'Max Volume Size': '' | 'CapacityStatus': 'Used' | 'Instance Capacity - 12xlarge': '' | 'Transfer Type': '' | 'Elastic GPU Type': '' | 'usageType': 'USE2-BoxUsage:t2.medium' | 'RateCode': '2N2QH6UEJZ5GUPT8.JRTCKXETXF.6YS6EN2CT7' | 'Instance Capacity - 24xlarge': '' | 'Instance Family': 'General purpose' | 'Currency': 'USD' | 'Enhanced Networking Supported': '' | 'serviceCode': 'AmazonEC2' | 'Physical Cores': '' | 'Instance Capacity - medium': '' | 'Volume Type': '' | 'Storage Media': '' | 'EndingRange': 'Inf' | 'Clock Speed': 'Up to 3.3 GHz' | 'From Location Type': '' | 'Unit': 'Hrs' | 'Current Generation': 'Yes'} |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| {'SKU': 'QT7848TA4YHDW5JE' | 'OfferingClass': '' | 'Group': '' | 'Instance Capacity - xlarge': '' | 'Instance Capacity - 16xlarge': '' | 'PricePerUnit': '0.0464000000' | 'PriceDescription': '$0.0464 per Unused Reservation Linux t2.medium Instance Hour' | 'Storage': 'EBS only' | 'Pre Installed S/W': 'NA' | 'Instance': '' | 'Normalization Size Factor': '2' | 'Location': 'US East (Ohio)' | 'Memory': '4 GiB' | 'Physical Processor': 'Intel Xeon Family' | 'operation': 'RunInstances' | 'Dedicated EBS Throughput': '' | 'Instance Capacity - 10xlarge': '' | 'Instance Capacity - 4xlarge': '' | 'To Location': '' | 'From Location': '' | 'Operating System': 'Linux' | 'Product Family': 'Compute Instance' | 'GPU': '' | 'Intel Turbo Available': '' | 'Intel AVX Available': '' | 'Max IOPS Burst Performance': '' | 'Instance Capacity - 32xlarge': '' | 'ECU': 'Variable' | 'Tenancy': 'Shared' | 'Instance Capacity - 18xlarge': '' | 'OfferTermCode': 'JRTCKXETXF' | 'Instance Capacity - 9xlarge': '' | 'Instance Capacity - 8xlarge': '' | 'Processor Architecture': '32-bit or 64-bit' | 'EBS Optimized': '' | 'Group Description': '' | 'Provisioned': '' | 'Location Type': 'AWS Region' | 'EffectiveDate': '2018-10-01' | 'License Model': 'No License required' | 'vCPU': '2' | 'TermType': 'OnDemand' | 'instanceSKU': '2N2QH6UEJZ5GUPT8' | 'PurchaseOption': '' | 'Instance Type': 't2.medium' | 'Instance Capacity - 2xlarge': '' | 'LeaseContractLength': '' | 'Instance Capacity - large': '' | 'StartingRange': '0' | 'Max IOPS/volume': '' | 'Max throughput/volume': '' | 'To Location Type': '' | 'Processor Features': 'Intel AVX; Intel Turbo' | 'Intel AVX2 Available': '' | 'GPU Memory': '' | 'serviceName': 'Amazon Elastic Compute Cloud' | 'Network Performance': 'Low to Moderate' | 'Max Volume Size': '' | 'CapacityStatus': 'UnusedCapacityReservation' | 'Instance Capacity - 12xlarge': '' | 'Transfer Type': '' | 'Elastic GPU Type': '' | 'usageType': 'USE2-UnusedBox:t2.medium' | 'RateCode': 'QT7848TA4YHDW5JE.JRTCKXETXF.6YS6EN2CT7' | 'Instance Capacity - 24xlarge': '' | 'Instance Family': 'General purpose' | 'Currency': 'USD' | 'Enhanced Networking Supported': '' | 'serviceCode': 'AmazonEC2' | 'Physical Cores': '' | 'Instance Capacity - medium': '' | 'Volume Type': '' | 'Storage Media': '' | 'EndingRange': 'Inf' | 'Clock Speed': 'Up to 3.3 GHz' | 'From Location Type': '' | 'Unit': 'Hrs' | 'Current Generation': 'Yes'} |
| {'SKU': 'PRCADQFUQ6HZKBHK' | 'OfferingClass': '' | 'Group': '' | 'Instance Capacity - xlarge': '' | 'Instance Capacity - 16xlarge': '' | 'PricePerUnit': '0.0000000000' | 'PriceDescription': '$0.00 per Reservation Linux t2.medium Instance Hour' | 'Storage': 'EBS only' | 'Pre Installed S/W': 'NA' | 'Instance': '' | 'Normalization Size Factor': '2' | 'Location': 'US East (Ohio)' | 'Memory': '4 GiB' | 'Physical Processor': 'Intel Xeon Family' | 'operation': 'RunInstances' | 'Dedicated EBS Throughput': '' | 'Instance Capacity - 10xlarge': '' | 'Instance Capacity - 4xlarge': '' | 'To Location': '' | 'From Location': '' | 'Operating System': 'Linux' | 'Product Family': 'Compute Instance' | 'GPU': '' | 'Intel Turbo Available': '' | 'Intel AVX Available': '' | 'Max IOPS Burst Performance': '' | 'Instance Capacity - 32xlarge': '' | 'ECU': 'Variable' | 'Tenancy': 'Shared' | 'Instance Capacity - 18xlarge': '' | 'OfferTermCode': 'JRTCKXETXF' | 'Instance Capacity - 9xlarge': '' | 'Instance Capacity - 8xlarge': '' | 'Processor Architecture': '32-bit or 64-bit' | 'EBS Optimized': '' | 'Group Description': '' | 'Provisioned': '' | 'Location Type': 'AWS Region' | 'EffectiveDate': '2018-10-01' | 'License Model': 'No License required' | 'vCPU': '2' | 'TermType': 'OnDemand' | 'instanceSKU': '2N2QH6UEJZ5GUPT8' | 'PurchaseOption': '' | 'Instance Type': 't2.medium' | 'Instance Capacity - 2xlarge': '' | 'LeaseContractLength': '' | 'Instance Capacity - large': '' | 'StartingRange': '0' | 'Max IOPS/volume': '' | 'Max throughput/volume': '' | 'To Location Type': '' | 'Processor Features': 'Intel AVX; Intel Turbo' | 'Intel AVX2 Available': '' | 'GPU Memory': '' | 'serviceName': 'Amazon Elastic Compute Cloud' | 'Network Performance': 'Low to Moderate' | 'Max Volume Size': '' | 'CapacityStatus': 'AllocatedCapacityReservation' | 'Instance Capacity - 12xlarge': '' | 'Transfer Type': '' | 'Elastic GPU Type': '' | 'usageType': 'USE2-Reservation:t2.medium' | 'RateCode': 'PRCADQFUQ6HZKBHK.JRTCKXETXF.6YS6EN2CT7' | 'Instance Capacity - 24xlarge': '' | 'Instance Family': 'General purpose' | 'Currency': 'USD' | 'Enhanced Networking Supported': '' | 'serviceCode': 'AmazonEC2' | 'Physical Cores': '' | 'Instance Capacity - medium': '' | 'Volume Type': '' | 'Storage Media': '' | 'EndingRange': 'Inf' | 'Clock Speed': 'Up to 3.3 GHz' | 'From Location Type': '' | 'Unit': 'Hrs' | 'Current Generation': 'Yes'} |
The difference in the three prices is the price description.
- 'PriceDescription': '$0.0464 per On Demand Linux t2.medium Instance Hour'
- 'PriceDescription': '$0.0464 per Unused Reservation Linux t2.medium Instance Hour'
- 'PriceDescription': '$0.00 per Reservation Linux t2.medium Instance Hour'
Basically, minion-manager currently only support on-demand instances (does not support Reserved instances). Therefore, only the "On Demand" price description from the above is relevant. But current implementation of the price querying API does not factor this in.
To start with:
- it'll be good to specifically look for "On Demand " in the price description and only consider that price.
- Add warnings if there are duplicates and if some price is being overwritten
- Ensure that the price is not set to 0. If so... log LOUDLY!!