optimize table fetch from bigquery
Closed this issue · 0 comments
bsushmith commented
Summary
Currently, in bigquery GetResources call, we do a mutex lock before we do append to the resources slice. But since we do a defer unlock, the entire code block is blocking other goroutines. So even though we have errgroup with limit of 20
, the code is running sequentially only due to the defer unlock.
This can be made faster by just locking and unlocking the specific non-concurrent code.