/DDIA

Primary LanguagePythonThe UnlicenseUnlicense

DDIA

Project

Data Model and Storage Engine Focus: Chapters 1-3 Task: Implement a simple key-value store with a basic storage engine. Skills: File I/O, data structures, serialization Data Encoding and Evolution Focus: Chapter 4 Task: Extend the key-value store to support multiple data formats (JSON, Protocol Buffers, Avro) and implement schema evolution. Skills: Data serialization, schema design, backward/forward compatibility Replication and Partitioning Focus: Chapters 5-6 Task: Transform the single-node key-value store into a distributed system with basic replication and partitioning. Skills: Network programming, consistency models, partition strategies Transactions and Consistency Focus: Chapter 7 Task: Implement multi-key transactions and explore different isolation levels in the distributed key-value store. Skills: Concurrency control, ACID properties, distributed transactions Distributed System Challenges Focus: Chapters 8-9 Task: Add fault tolerance, consensus algorithms, and batch/stream processing capabilities to the distributed key-value store. Skills: Leader election, consensus protocols, batch and stream processing Derived Data and Analytics Focus: Chapters 10-12 Task: Implement a data pipeline that uses the distributed key-value store as a source, performs ETL operations, and loads data into an analytics-focused data store (e.g., a column-oriented store). Skills: ETL processes, OLAP vs OLTP, data warehousing concepts