zshamrock/dynocsv

Support read rate by reading table/index RCU, and adjust the consumption accordingly

Opened this issue · 2 comments

Also, it will not hit up to the max of the RCU configured, but only up to 80% of it.

Currently tried the approach of limiting the number of ms per item, by using the following formula:

ms per item = 1000 / RCU / 2, and then accumulate the elapsed time as total items * ms per item, if it is greater than now, sleep for the difference between elapsed - now ms.

Although even this didn't push the table up to its capacity limit.

Worth to try the alternative approach where the consumed capacity is returned back in the response, and also sleep, but sleep time is determined by the allowed capacity for elapsed time, i.e. table's RCU per second, and actually consumed capacity so far. This should be a more correct approach, as ms per item doesn't count for the data size in bytes, where 1 RCU is

One read capacity unit represents one strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size.

as per https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html