/klein

🔥 Klein is a Paxos based distributed collection tool library, including distributed ArrayList, distributed HashMap, distributed Cache, distributed Lock, etc..

Primary LanguageJavaApache License 2.0Apache-2.0

中文

Thanks to JetBrains for the free license.

Open source： Gitee | Github | CodeChina

     

Introduce

logo

Klein is a Paxos based distributed collection tool library, including distributed ArrayList, distributed HashMap, distributed Cache, distributed Lock, etc..

In addition, based on Paxos, we can imagine endlessly, such as: KV-storage, Registry, Configuration center, etc.

We hope that Klein can be deployed independently or embedded in your project. You can use it just like Redis, or embed Klein into your project without relying on any middleware to ensure data consistency among members. Of course, this is still in the process of implementation. 😆😆😆

Look forward to your star⭐

Quick Start

dependence klein

<dependency>
    <groupId>com.ofcoder.klein.core</groupId>
    <artifactId>klein-core</artifactId>
    <version>0.0.1-SNAPSHOT</version>
</dependency>
<repositories>
    <repository>
        <id>ossrh</id>
        <url>https://s01.oss.sonatype.org/content/repositories/snapshots</url>
    </repository>
</repositories>

startup

Klein instance = Klein.startup();
instance.getCache().put("hello", "klein");

set property

For all configurable parameters, see: com.ofcoder.klein.KleinProp

You can set it through System#setProperty or get the KleinProp object

System.setProperty("klein.id", "2")

// or

KleinProp prop = KleinProp.loadIfPresent();

Milepost

Evolve

Paxos

  • Write request, disordered negotiation, sequential confirmation
  • Read request, using negotiation log
  • Batch negotiation
  • Optimize the prepare phase
  • Snapshot
  • To split roles such as Group and proposer, you only need to isolate instance instead of isolating them
  • Master role：
    • Change of members
    • The master promotion should have the most complete data (the master should be elected through negotiation with the proposal. If the promotion is successful, then boost did not reach a consensus)
    • Keep data consistent
      • Master heartbeat triggers data synchronization
      • Snapshot synchronization (the heartbeat carries the checkpoint and the learn message returns the checkpoint)
      • New members join the cluster and actively learn from the master
    • Optimize read requests (write requests must be copied to the master)
    • Optimize write requests (write requests can only be executed by the master to avoid livelocks)
  • Automatic member discovery (research)
  • NWR
  • Verified by jepsen
    • Linearly consistent read and write
    • Network partition
    • Member outage

Cache

  • Basic functions such as reading, writing, etc
  • Implement LRU with persistence
  • Cache Automatic Expiration (TTL)
  • Clock skew

Collection

  • list
  • map

To be optimized

  • LogManager row lock
  • Monitor negotiation efficiency
  • Monitoring thread pool indicators (DefaultTimer, ThreadExecutor)
  • ProposalNo全局唯一
  • 状态机持久化（master、lock）

Design ideas

Paxos

  • How to generate ProposalNo?
  • Can parallel negotiation really be supported?
  • Which proposal will reach consensus?
  • Can the Confirm phase (application state transition) really be executed asynchronously?
  • How do I create a snapshot of a running system?
  • Is it necessary to completely isolate the splitting of a group?
  • Optimize the Prepare phase
  • Batch negotiation (queue) to reduce RPC interaction

Star History

Star History Chart