This repository contains list of distributed system projects with open source code in various programming languages which may be useful in order to better understand how to build distributed services.
- (Golang) Jocko - Kafka/distributed commit log service in Go. [Serf + Raft]
- (Golang) oklog - distributed and coordination-free log management system for big ol' clusters [Archived]
- (Golang) Kingbus - a distributed MySQL binlog store based on raft [Raft]
- (Golang) elasticell - a distributed HA Redis-compatible NoSQL database with strong consistency and reliability
- (Golang) RQLite - lightweight, distributed relational database, which uses SQLite as its storage engine
- (Golang) CockroachDB - distributed fault-tolerant SQL database built on a transactional and strongly-consistent key-value store
- (Erlang) CouchDB - highly available, partition tolerant, eventually consistent document database . Supports master-master setups with automatic conflict detection.
- (C++) MongoDB - document database designed for ease of development and scaling
- (Java) Apache Cassandra - a highly-scalable partitioned row store. Rows are organized into tables with a required primary key
- (Java) Apache HBase - Hadoop database, a distributed, scalable, big data store. Useful when random, realtime read/write access to big data needed
- (Rust) TiKV - distributed transactional key-value database, originally created to complement TiDB
- (Golang) Tair - high-performance and high-availability distributed fast-access memory (MDB)/persistent (LDB) storage service
- (C) memcached - high performance multithreaded event-based key/value cache store intended to be used in a distributed system
- (Golang) etcd - distributed reliable key-value store for the most critical data of a distributed system [Raft + gRPC]
- (Java) Apache Zookeeper - highly reliable distributed coordination
- (Golang) chubby - A (very simplified) implementation of Chubby, Google's distributed lock service
- (Java) Kafka - distributed, highly scalable, elastic, fault-tolerant, and secure event streaming platform
- (Python) faust - a distributed stream processing library that ports the ideas from Kafka Streams to Python
- (Golang) Liftbridge - lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system
- (Golang) dkron - distributed, fault tolerant job scheduling system for cloud native environments
- (Python) Celery - distributed task queue
- (Python) Apache Airflow - a platform to programmatically author, schedule, and monitor workflows
- (Golang) nsq - realtime fault tolerant distributed messaging platform designed to operate at scale, handling billions of messages per day [Raft + gRPC]
- (Golang) Sandglass - distributed, horizontally scalable, persistent, time ordered message queue
- (Golang) dnpipes - distributed version of Unix named pipes comparable to AWS SQS
- (PHP) GatewayWorker - distributed realtime messaging framework based on workerman
- (C++) ZeroMQ - abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more
- (Java) Apache Pulsar - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API
- (Java) Apache ActiveMQ - high performance Apache 2.0 licensed Message Broker
- (Java) ElasticSearch - distributed, RESTful search and analytics engine
- (Java) Apache Lucene - a high-performance, full featured text search engine library
- (Rust) MeiliSearch - Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine
- (JS) FlexSearch - memory-flexible full-text search library
- (Golang) RiotSearch - distributed, Simple and efficient full text search engine
- (C++) Typesense - fast, typo tolerant, fuzzy search engine
- (Golang) JuiceFS - Hadoop-compatible AWS S3-compatible high-performance POSIX file system
- (Golang) SeaweedFS - a simple Hadoop-compatible AWS S3-compatible distributed highly scalable distributed file system
- (C) GlusterFS - distributed storage that can scale to several petabytes
- (C++) GlusterFS - highly reliable, scalable and efficient distributed file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.
- (Golang) etcd - framework for distributed systems development. Provides the core requirements for distributed systems development including RPC and Event driven communication
- (Golang) ergo - port of Erlang/OTP approaches in Golang
- (Golang) gosiris - an actor framework for Golang
- (Python) cotyledon - a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more.
- (Java) atomix - fully featured framework for building fault-tolerant distributed systems [REST + Raft]
- (Kotlin) orbit - virtual actor framework for building distributed systems
- (JS) hemera - A Node.js microservices toolkit for the NATS messaging system [RPC]
- (Python) Tooz - centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications
- (C++) Nebula - powerful framework for building highly concurrent, distributed, and resilient message-driven applications
- (Golang) Dragonboat - a high performance multi-group Raft consensus library in pure Go
- (Golang) Golimit - Uber ringpop based distributed and decentralized rate limiter
- (Python) Tenacity - general-purpose retrying library
- (Elixir) ex_hash_ring - pure Elixir consistent hash ring implementation based on the excellent C hash-ring lib
- (Elixir) raft - Raft consensus implementation
- (C++) NuRaft - Raft implementation derived from the cornerstone project