If you appreciate the content 📖, support projects visibility, give 👍| ⭐| 👏
A curated list of awesome System Designing articles, videos and resources for distributed computing, AKA Big Data.
Whether you're preparing for an interview or you want to design a distributed/microservice oriented application, this list will definitely help you achieve that.
Attention: Stars on GitHub does not reflect usage or popularity for every item here listed.
Inspired By Awesome-BigData
Started By Gabriel Leon de Mattos
-
System Design Primer - [109k ⭐] - Awesome compilation of resources, including Anki flashcard decks.
-
System Design Interview Questions - Concepts you should know - A curated list of topics to introduce you to system design.
-
Grokking the System Design Interview - [Paid 💵] - Grokking System Design preparation is one of the most talked about course. The best thing about it is the design of applications it suggests rather than explanations of what each tool is supposed to do.
-
System Design in Software Development - Basic article on the topics of system design and architecture.
-
Everything you need to know about Design System - Another introductory article written in a friendly way.
-
System Design - Introductory interview preparation resources.
-
Design Pattern for Distributed Systems - Article talking about some patterns as well as some technologies to be considered.
-
Distributed Computing - Wikipedia article broadening the view of distributed system design.
-
Fallacies of Distributed Computing - Wikipedia article introducing the topic of fallacies of distributed computing and its effects.
-
Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
-
CAP Theorem - IBM Article about CAP Theorem, Microservices and NoSQL DBs.
-
Pattern: Microservice Architecture - Good article talking about Microservice architecture as well as its drawbacks.
-
Taxonomy of Distributed Systems - 11 Page lecture classifying distributed systems and specifically why we need them.
-
Top 10 Secure Coding Practices - Brief article talking about good practices for code securities.
-
Scalable Web Architecture and Distributed Systems - Good article about distributed systems as well as some of the potential tools.
-
Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid 💵] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.
-
Designing Data Intensive Applications - [Paid 💵] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.
-
Building Microservices - [Free 👍] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.
-
Monolith to Microservices - [Free 👍] - Written by the same author as the one above, this book will cover the migration from Monolith to Microservices, it's recommended you start by the previous book.
A collection of videos based on distributed systems.
-
Gaurav Sen - System Design Series - Good resource for people who want to learn more about system design, introduces the topic in a very easy to understand way.
-
Tech Dummies - System Design Series - Another introduction to system design.
-
Mock System Design Interview at Google - Overview of what an interview on system design would look like from the perspective of a flawed but close fulfilling of the requirements. Key thing here is how the interaction with the interviewer goes.
-
Google Preparation Guide - A quick video explaining how they interview.
- The evolution of Reddit Architecture - Overview of how Reddit system design scaled.
- A collection of most commonly used tools for distributed systems
-
MariaDB - MariaDB is a fork of MySQL server.
-
MySQL - Widely used relational database.
-
PostgresSQL - Relational database that has been gaining popularity.
-
SQLite - Another widely used database that is built into all mobile phones and most computers.
-
Apache Ignite - [3.3k ⭐] - In memory caching with ACID properties.
-
Couchbase - Inspired by memcached, adding features such as replication and persistance.
-
Oracle Coherence - [126 ⭐] - High scaling, low latency in-memory caching.
-
Memcached - [10.2k ⭐] - One of the first in-memory caching database, high performing and multi-threaded.
-
Redis - [44k ⭐] - Widely used in-memory caching database with many added features such as persistent storage and supporting strings, lists, sets, hashses, streams, bitmaps, etc.
- Apple FoundationDB - [10k ⭐] - Multi-model (many data types in a single database), ACID key-value store. Easily scalable and fault tolerant.
-
CouchDB - [4.6k ⭐] - ACID compliant NoSQL document-store DB, provides a RESTful HTTP API for reading and updating database documents.
-
MongoDB - One of the most popular 'NoSQL' database for general purpose.
-
RethinkDB - [23.8k ⭐] - Document-store DB.
-
ElasticSearch - [49.9k ⭐] - Widely popular 'NoSQL' database for fast and scalable search engines.
-
Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.
-
Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.
-
Cassandra - Facebook-born project very fast, easily scalable, with option to include consistency with each operation.
-
Scylla - [4.9k ⭐] - 'NoSQL' data store using seastar framework, compatible with Cassandra.
-
HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project and runs on top of HDFS or Alluxio. (See Hadoop Related)
-
Amazon Neptune - Fast, reliable and fully managed graph database service.
-
ArangoDB - [10k ⭐] - Flexible database for documents, key-value, graphs. Uses its own query language, AQL.
-
Neo4j - [7.9k ⭐] - Good support for a graph db, ACID compliant and flexible.
-
HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.
-
Lustre - File system for computer clusters.
-
CephFS - Unified, distributed storage system.
-
GlusterFS - Scale-out NAS file system.
-
MooseFS - POSIX-compliant distributed file system.
-
XtreemFS - Fault tolerant file system.
- Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.
-
Amazon MQ - Open source message broker from Amazon.
-
Apache ActiveMQ - It's a multi-protocol, java based messaging server.
-
Apache Kafka - Widely popular message broker with low latency for data streaming.
-
RabbitMQ - Widely popular lightweight message broker written in erlang that also supports multiple messaging protocols.
-
IronMQ - Very fast and highly scalable messaging broker. (not open source)
-
Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.
-
Kestrel - Written in Scala and speaks the memcached protocol. It works much like Kafka. Celery Cloverleaf (E-Novation Lifeline)
-
SeeSaw - [5.1k ⭐] - Used by Google, developed in Go, linux-based virtual load balancer server.
-
HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.
-
Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
-
Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.
-
Ngnix - Wait, isn't Nginx a web server? Yes, the open source does support basic level of content switching and request routing. Plus edition supports load balancing, WAF, monitoring, etc.
-
F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).
-
TP-Link - Cheaper alternative that works as a load balancer.
-
Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.
-
Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.
-
Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.
-
Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
-
DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.
-
Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.
- Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.
- Hive - Query hadoop stored data in SQL.
- Pig - Scriping language that looks like SQL to query hadoop data.
- Tez - Solves a similar problem to Spark and MapReduce, it's more efficient than MapReduce because it calculates the most efficient way of doing it.
- Map Reduce - MapReduce, as the name implies, maps data and reduce the results.
- Spark - Powerful data processing to not only process data like Tez (and MapReduce), it can process streams of data in real time, apply regression analysis algorithms in ML and much more.
- HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project.
- YARN - 'Yet Another Resource Negotiator', works like a kernel to manage computer resources across the clusters.
- MESOS - Works like a Linux Kernel by managing CPU, memory, storage and other resources across the cluster.
- Gin - [40.6k ⭐] - Blazingly fast microservice framework using Golang, high throughput capacity.
- Phoenix - [15.5k ⭐] - Distributed processing, easily scalable, support for channels and live chat. This framework uses BEAM and Erlang, very efficient for large scale systems and supports high throughput.
- Express.js - [49.6k ⭐] - Fast node.js rest api that can perform well under many scenarios.
- Rails - [46.2k ⭐] - Written in Ruby, Rails delivers quick apis from prototype to production in an efficient manner.
- Play Framework - [11.6k ⭐] - Very fast, high throughput framework written in Scala/Java that is RESTful by default.