Use a better `ReliableStorageLock` based on If-Match S3 atomic operations
IvoDD opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
The current ReliableStorageLock
was being implemented before If-Match
on put objects was announced by aws and hence uses a more complex structure than it is needed. This ticket lists all minor things we need to address when doing this simplification.
This is blocked on aws-sdk-cpp not having 1.11.455 or later which has the new If-Match on put requests support.
Describe the solution you'd like
- Use lock which just relies on a single ref key which gets overriden with If-Match statements.
- Think about how existing uses of the lock can upgrade to the new locking
- Rework C++ ReliableStorageLock tests to use a
MockS3Client
instead of theInMemoryStore
. more info - Make the storage lock configure the underlying storage with a
MaxTimeoutPoliocy
. more info - Use Last-Modified time from S3 for expiration instead of the process time. more info
- Move slower cpp stress tests to a separate target. more_info
Also we need to think about:
- Dynamically detecting whether storage supports atomic operations
- Provide a means to fallback to the old storage lock (which will need some new APIs e.g. extending a lock)
Rework C++ ReliableStorageLock tests to use a MockS3Client instead of the InMemoryStore. #2014 (comment)
Attempting to write such python tests turned out futile because of the GIL handling. It would probably still be better to have the C++ tests use MockS3Client
instead of InMemoryStore