/Distributed-Key-Value-storage-Proj

Primary LanguageGo

I can't directly create files on your system, but I can rewrite the readme for you in more detail:

minikeyvalue: A Simple Distributed Key-Value Store

Minikeyvalue is a lightweight (around 1000 lines of code) distributed key-value store written in Rust. It's designed for simplicity and scalability, making it a good option for storing large amounts of data (between 1MB and 1GB per value) across multiple machines.

Inspired by SeaweedFS, minikeyvalue offers several key advantages:

  • Simple design: It leverages standard tools like Nginx for volume serving and LevelDB for indexing, making it easy to understand and maintain.
  • Scalability:
    • Supports replication to ensure data redundancy and availability.
    • Can handle multiple machines and multiple storage drives per machine.
    • Designed to scale to billions of files and petabytes of data.
  • Production-ready: Used in production at comma.ai, demonstrating its reliability.

Here's a breakdown of how minikeyvalue works:

  • Storage:
    • Data is stored on the filesystem using Nginx as a volume server.
    • LevelDB is used to keep track of where data is located.
  • Key-Value API:
    • Supports basic CRUD (Create, Read, Update, Delete) operations on key-value pairs.
    • Uses standard HTTP methods (GET, PUT, DELETE) for interaction.
    • Partially compatible with a subset of S3 requests for some S3 libraries.
  • Rebalancing and Rebuilding:
    • Volumes (storage servers) can be added or removed to adjust storage capacity.
    • The LevelDB index can be rebuilt if necessary.

Getting Started:

  1. Start Volume Servers: Run the volume binary on each machine that will store data, specifying a port and storage directory.
  2. Start Master Server: Run the mkv binary with the server flag, specifying the LevelDB database path, volume server addresses (comma-separated), and desired options (port, replicas, etc.).

Using the API:

  • You can use tools like curl to interact with the API.
  • Refer to the readme for specific commands for various operations (e.g., putting data, getting data, deleting data).

Rebalancing and Rebuilding:

  • The readme provides instructions on how to rebalance volumes (change the number of storage servers) and rebuild the LevelDB index.

Performance:

The readme also includes some performance benchmarks for fetching nonexistent keys.

Additional Notes:

  • The provided project description seems to be referring to a separate project, not directly related to minikeyvalue.

I hope this detailed explanation helps you understand minikeyvalue better!