roma-glushko/awesome-distributed-system-projects

🚀 List of distributed system projects for inspiration and learning to build distributed services from real world examples

Awesome Distributed Systems

This repository contains list of distributed system projects with open source code in various programming languages which may be useful in order to better understand how to build distributed services.

Databases

(Golang) Jocko - a Kafka/distributed commit log service in Go. [Serf + Raft]
(Golang) oklog - a distributed and coordination-free log management system for big ol' clusters [Archived]
(Golang) elasticell - a distributed HA Redis-compatible NoSQL database with strong consistency and reliability
(Erlang) CouchDB - a highly available, partition tolerant, eventually consistent document database . Supports master-master setups with automatic conflict detection.
(Java) Apache HBase - a Hadoop database, a distributed, scalable, big data store. Useful when random, realtime read/write access to big data needed
(Golang) Tair - a high-performance and high-availability distributed fast-access memory (MDB)/persistent (LDB) storage service
(Golang) immudb - an immutable database based on zero trust, Key/Value & SQL, tamperproof, data change history
(Rust) toydb - distributed SQL database in Rust, written as a learning project
(Rust) DB3 Network - a decentralized firebase firestore alternative
(Python) ZODB - an ACID transactional object-oriented database
(Golang) requiemdb - a permanent storage for OTEL data

Key-Value Databases

(C) memcached - a high performance multithreaded event-based key/value cache store intended to be used in a distributed system
(C) redis - an in-memory database with various value types that persists on disk
(Rust) TiKV - a distributed transactional key-value database, originally created to complement TiDB
(C++) leveldb - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values
(Golang) goleveldb - a LevelDB implemented in Golang
(Golang) summitdb - an in-memory, NoSQL key/value database. It persists to disk, uses the Raft consensus algorithm, is ACID compliant, and built on a transactional and strongly-consistent model.It supports custom indexes, geospatial data, JSON documents, and user-defined JS scripting
(Python) pupdb - a simple file-based key-value database
(Python) pickledb - an open source key-value store using Python's json module
(C++) KeyDB - a faster drop-in multithreaded alternative to Redis
(C++) Dragonfly - an in-memory data store fully compatible with Redis and Memcache and designed using modern algorithms
(Golang) BadgerDB - an embeddable, persistent and fast key-value (KV) database written in pure Go
(Golang) BuntDB - a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports custom indexes and geospatial data.
(Rust) ConstDB - a redis-like cache store that implements CRDTs and active-active replications.
(Golang) GhostDB - a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale
(Dart) Hive - a lightweight and blazing fast key-value database written in pure Dart. Inspired by Bitcask
(Golang) rosedb - a fast, stable, and embedded NoSQL database based on bitcask, supports a variety of data structures such as string, list, hash, set, and sorted set
(Rust) PumpkinDB - an immutable Ordered Key-Value Database Engine
(Golang) FlashDB - a simple, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports redis like operations for data structures like SET, SORTED SET, HASH and STRING
(PHP) Lazer - a PHP flat file database based on JSON files
(Golang) Scribble - a tiny JSON database in Golang
(Golang) FlyDB - a high-performance KV storage engine based on bitcask paper supports redis protocol and the corresponding data structure
(Rust) Engula - a distributed key-value store, used as a cache, database, and storage engine
(Golang) Dice - an extremely simple Golang-based in-memory KV store that speaks Redis dialect

Relational, SQL, NewSQL Databases

NoSQL, Document Databases

Graph Databases

Time Series

Column Databases

Permission Databases

Analytical Databases

Vectors

Gateways

(Golang) Glide - an open reliable fast LLM/model gateway for rapid development of GenAI apps
(Golang) Traefik - a cloud-native app proxy
(Lua) Kong - a cloud-native feature-rich API gatewat
(Golang) Skipper - an HTTP router and reverse proxy for service composition
(Golang) janus - a lightweight API gateway and management platform
(Golang) Lura - ultra performance API gateway with middlewares
(Python) MLFLow Gateway - an LLM proxy

Locking

Streaming

Schedulers

Queues

Search Engines

File Systems

(Golang) JuiceFS - Hadoop-compatible AWS S3-compatible high-performance POSIX file system
(Golang) SeaweedFS - a simple Hadoop-compatible AWS S3-compatible distributed highly scalable distributed file system
(C) GlusterFS - distributed storage that can scale to several petabytes
(C++) GlusterFS - highly reliable, scalable and efficient distributed file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.

Service Discovery

(Golang) sleuth - master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network

Data Processing

(Scala) Apache Spark - unified analytics engine for large-scale data processing

TerminusDB

(Prolog) terminusdb - distributed database with a collaboration model

OS

(C) HarveyOS - distributed operating system

Frameworks

(Golang) etcd - framework for distributed systems development. Provides the core requirements for distributed systems development including RPC and Event driven communication
(Golang) ergo - port of Erlang/OTP approaches in Golang
(Golang) gosiris - an actor framework for Golang
(Python) cotyledon - a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more.
(Java) atomix - fully featured framework for building fault-tolerant distributed systems [REST + Raft]
(Kotlin) orbit - virtual actor framework for building distributed systems
(JS) hemera - A Node.js microservices toolkit for the NATS messaging system [RPC]
(Python) Tooz - centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications
(C++) Nebula - powerful framework for building highly concurrent, distributed, and resilient message-driven applications
(GoLang) Service Weaver - A framework that allows to write applications as modular binary and deploy it as a set of microservices
(GoLang) Dapr - portable, serverless, event-driven runtime that works as a sidecar and makes it easy for developers to build resilient, stateless and stateful microservices

Components

Other Resources