A from-scratch implementation of Kafka's core functionality in Python, using only the standard library. This project aims to replicate Kafka's message brokering capabilities and protocol, providing a learning experience and a potentially lightweight alternative for specific use cases.
- Kafka Protocol Compatibility: Adheres to the Kafka protocol for communication, ensuring interoperability with Kafka clients.
- APIVersion Handling: Handles APIVersion requests from clients.
- Fetch Command (In Progress): Actively working on implementing the
Fetchcommand for message consumption.
- Message Production and Consumption: Implement basic message production and consumption functionality.
- Topic Management: Support topic creation and management.
- Complete Fetch Implementation: Add support for concurrent requests, consuming messages from disk, and handling various Fetch scenarios.
- Persistence: Implement persistent storage for messages, ensuring durability.
- Consumer Groups and Rebalancing: Support consumer groups for scalable message consumption and implement rebalancing for fault tolerance.
- Replication: Add support for replication to ensure data availability and fault tolerance.
- Performance Optimization: Optimize message handling and storage for improved performance.
- Clone the Repository:
git clone [invalid URL removed]- Install Dependencies:
Ensure you have Python 3.8 and higher installed.
pip install ruff- Run the Server:
python -m app.main-
Contributing Contributions are welcome! Feel free to open issues or submit pull requests. Please adhere to the project's coding style and ensure your code passes Ruff checks.
-
Disclaimer This project is a work in progress and is not intended for production use. It serves as a learning exercise and a potential foundation for further development.
-
License This project is licensed under the MIT License.