qdrant/qdrant

Implement S3 snapshot manager

Opened this issue · 11 comments

generall commented

Is your feature request related to a problem? Please describe.

Continuation of the issue #3324

In previous iterations we decomposed Snapshot manager into multiple implementations, depending of the storage backend.

qdrant/lib/collection/src/common/snapshots_manager.rs

Line 192 in 61fa4e7

impl SnapshotStorageS3 {

But SnapshotStorageS3 is currently not implemented and not used.

Describe the solution you'd like

  • Implement missing functions in SnapshotStorageS3
  • Alter the settings, to allow explicitly select between local FS and S3
    Example:
storage:

  # Where to store snapshots
  snapshots_path: ./snapshots
  
  s3_config: 
    bucket: "s3://hello-world"
  
  snapshots_storage: "s3" | "local" # "local" - default

Describe alternatives you've considered

Additional context

There are some notable previous attempts which might be helpful - #3430

generall commented

/bounty $200

algora-pbc commented

## 💎 $200 bounty • Qdrant

### Steps to solve:
1. Start working: Comment /attempt #4109 with your implementation plan
2. Submit work: Create a pull request including /claim #4109 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

### Additional opportunities:
- 🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Comment /livestream once live

Thank you for contributing to qdrant/qdrant!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🔴 @varshith257 Apr 25, 2024, 8:18:54 PM WIP
🟢 @kemkemG0 Apr 29, 2024, 10:07:24 PM #4150
🔴 @kid-116 Apr 30, 2024, 7:26:40 PM WIP
🟢 @abhishek818 May 1, 2024, 4:01:36 AM WIP
varshith257 commented

@generall I would like to solve this issue with your guidance though I am new to this repo :)
/attempt #4109

Options
varshith257 commented

What are the expected functions to fully implement SnapshotStorageS3?

generall commented 
impl SnapshotStorageS3 {
    async fn delete_snapshot(&self, _snapshot_path: &Path) -> CollectionResult<bool> {
        unimplemented!()
    }

    async fn list_snapshots(
        &self,
        _directory: &Path,
    ) -> CollectionResult<Vec<SnapshotDescription>> {
        unimplemented!()
    }

    async fn store_file(
        &self,
        _source_path: &Path,
        _target_path: &Path,
    ) -> CollectionResult<SnapshotDescription> {
        unimplemented!()
    }

    async fn get_stored_file(
        &self,
        _storage_path: &Path,
        _local_path: &Path,
    ) -> CollectionResult<()> {
        unimplemented!()
    }
}
  • 🚀2
  • 👀2
kemkemG0 commented

Let me give it a shot as well : )

/attempt #4109

Options
varshith257 commented

@kemkemG0 I am almost done with it. Can you please look at another issue rather waste of efforts? :)

kid-116 commented

/attempt #4109

Options
abhishek818 commented

/attempt #4109

Algora profile Completed bounties Tech Active attempts Options
@abhishek818    1 Qdrant bounty
+ 2 bounties from 2 projects
JavaScript, TypeScript
Cancel attempt
algora-pbc commented

💡 @kemkemG0 submitted a pull request that claims the bounty. You can visit your bounty board to reward.

algora-pbc commented

🎉🎈 @kemkemG0 has been awarded $200! 🎈🎊