CDF (Continuous Data Framework)

Craft end-to-end data pipelines and manage them continuously

📖 Table of Contents

📖 Table of Contents
📍 Overview
📦 Features
🚀 Getting Started
📚 Documentation
🤝 Contributing
🛣 Roadmap
📄 License
👏 Acknowledgments

📍 Overview

CDF (Continuous Data Framework) is an integrated framework designed to manage data across the entire lifecycle, from ingestion through transformation to publishing. It is built on top of two open-source projects, sqlmesh and dlt, providing a unified interface for complex data operations. CDF simplifies data engineering workflows, offering scalable solutions from small to large projects through an opinionated project structure that supports both multi-workspace and single-workspace layouts.

Warning

The repo is currently under ACTIVE development with multiple large refactors already having been completed. As such, you must be aware that the codebase is not yet stable and is subject to change. Furthermore, you must look to the code (or tests) itself for the most accurate and up-to-date information until this disclaimer is removed.

Features

...

Getting Started

Installation:

CDF requires Python 3.9 or newer. Install CDF using pip:
```
pip install python-cdf
```

Documentation

For detailed documentation, including API references and tutorials, visit CDF Documentation.

Contributing

Contributions to CDF are welcome! Please refer to the contributing guidelines for more information on how to submit pull requests, report issues, or suggest enhancements.

License

CDF is licensed under Apache 2.0 License.

This README provides an overview of the CDF tool, highlighting its primary features, installation steps, basic usage examples, and contribution guidelines. It serves as a starting point for users to understand the capabilities of CDF and how it can be integrated into their data engineering workflows.

🧪 Tests

Run the tests with pytest:

pytest tests

🛣 Project Roadmap

TODO: Add a roadmap for the project.

🤝 Contributing

Contributions are welcome! Here are several ways you can contribute:

Submit Pull Requests: Review open PRs, and submit your own PRs.
Join the Discussions: Share your insights, provide feedback, or ask questions.
Report Issues: Submit bugs found or log feature requests for z3z1ma.

Contributing Guidelines

Click to expand

Fork the Repository: Start by forking the project repository to your GitHub account.
Clone Locally: Clone the forked repository to your local machine using a Git client.
```
git clone <your-forked-repo-url>
```
Create a New Branch: Always work on a new branch, giving it a descriptive name.
```
git checkout -b new-feature-x
```
Make Your Changes: Develop and test your changes locally.
Commit Your Changes: Commit with a clear and concise message describing your updates.
```
git commit -m 'Implemented new feature x.'
```
Push to GitHub: Push the changes to your forked repository.
```
git push origin new-feature-x
```

7a. Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.

Once your PR is reviewed and approved, it will be merged into the main branch.

📄 License

This project is distributed under the Apache 2.0 License. For more details, refer to the LICENSE file.

👏 Acknowledgments

Harness (https://harness.io/) for being the proving grounds in which the initial concept of this project was born.
SQLMesh (https://sqlmesh.com) for being a foundational pillar of this project as well as the team for their support, advice, and guidance.
DLT (https://dlthub.com) for being the other foundational pillar of this project as well as the team for their support, advice, and guidance.

Return

z3z1ma/cdf