EC2 DescribeInstances API calls timeout
psypuff opened this issue · 1 comments
psypuff commented
Using dynamic host sets (AWS specifically, haven't tried the Azure one yet) in a large environment results in the following error:
{"id":"<REDACTED>","source":"https://hashicorp.com/boundary/4466e4d3c372/controller","specversion":"1.0","type":"error","data":{"error":"rpc error: code = Unknown desc = error running DescribeInstances for host set id \"<REDACTED>\": SerializationError: failed decoding EC2 Query response\n\tstatus code: 200, request id: 55f65e01-476e-4b4d-93b0-9e38b419b445\ncaused by: context deadline exceeded (Client.Timeout or context cancellation while reading body)","error_fields":{},"id":"<REDACTED>","version":"v0.1","op":"plugin.(SetSyncJob).syncSets","info":{"catalog id":"<REDACTED>","msg":"listing hosts"}},"datacontentype":"application/cloudevents","time":"2022-06-21T08:40:33.258736566Z"}
We tried splitting it into multiple host sets with different filters, still some queries timed out.
The biggest dynamic host sets we managed to produce were around 300 hosts. Some queries should've returned less hosts but timed out as well so host count might not be the only factor here.
Looking at the code it seems the DescribeInstances call uses the default HTTP client with the default timeout config. We're suggesting exposing some config to control the timeout of such calls.
Boundary version: 0.9
Setup: Docker on EC2 VMs