NikhilGeorge/odd-platform

First open-source data discovery and observability platform. ODD Platform is based on ODD Specification.

 

Open Data Discovery Platform: Next-Gen Data Discovery and Observability

Introduction

ODD is an open-source data discovery and observability tool for data teams that helps to efficiently democratise data, power collaboration and reduce time on data discovery through modern user-friendly environment.

Key wins

• Shorten data discovery phase

• Have transparency on how and by whom the data is used

• Foster data culture by continuous compliance and data quality monitoring

• Accelerate data insights

• Know the sources of your dashboards and ad hoc reports

• Deprecate outdated objects responsibly by assessing and mitigating the risks

• 👉 ODD Platform is a reference implementation of Open Data Discovery Spec.

Features

Data Discovery and Observability

• Accumulate scattered data insights in Federated Data catalogue
• Gain observability through E2E Data objects Lineage
• Benefit from cutting-edge E2E microservices Lineage feature in tracking your data flow through the whole data landscape
• Be warned and alerted by Pipeline Monitoring tools
• Store your metadata
• Use ODD-native modern lightweight UI

ML First citizen

• Save results of your ML Experiments by automatically logging its parameters

Data Security & Compliance

• Manage Tags and Labels to prevent any abuse of the data
• Refer to Tags and Labels to stay compliant with data security standards
• Have full transparency on how and by whom the data is used

Data Quality

• Simplify DQ processes by using ODD and Great Expectations compatibility
• Integrate ODD with any custom DQ framework

Getting Started

Running Locally with Docker Compose

docker-compose -f docker/demo.yaml up -d odd-platform-enricher

• 👉 QUICKSTART

Deploying to Kubernetes with Helm Charts

• 👉 QUICKSTART

Example configurations

There are various example configurations (via docker-compose) within docker/examples directory.

Contributing

Contributing to ODD Platform is very welcome. For basic contributions, all you need is being comfortable with GitHub and Git. The best ways to contribute are:

• Work on new adapters
• Work on documentation

To ensure equal and positive communication, we adhere to our Code of Conduct. Before starting any interactions with this repository, please read it and make sure to follow.

Please before contributing check out our Contributing Guide and issues labeled "good first issue":

Integrations

ODD Platform works with many of the tools you're already using:

Existing integrations
	Airflow
	Athena
	Cassandra
	Clickhouse
	DBT
	DynamoDB
	Elasticsearch
	Feast
	Glue
	Hive
	Kafka
	Kinesis
	Kubeflow
	Microsoft ODBC
	MongoDB
	MSSQL
	MySQL
	Neo4j
	PostgreSQL
	Quicksight
	Redshift
	S3
	SageMaker
	SageMaker Featurestore
	Snowflake
	Spark
	SQS
	Tableau
	Tarantool
	BigQuery
	Cube
	Vertica
	Trino
	Presto
	MariaDB
	SuperSet

ODD Data Model

ODD operates the following high-level types of entities:

1. Datasets (collections of data: tables, topics, files, feature groups)
2. Transformers (transformers of data: ETL or ML training jobs, experiments)
3. Data Consumers (data consumers: ML models or BI dashboards)
4. Data Quality Tests (data quality tests for datasets)
5. Data Inputs (sources of data)
6. Transformer Runs (executions of ETL or ML training jobs)
7. Quality Test Runs executions of data quality tests

For more information, please check specification.md.

Contacts

If you have any questions or ideas, please don't hesitate to drop a line to any of us.

Team Member	LinkedIn	GitHub
German Osin	LinkedIn	germanosin
Nikita Dementev	LinkedIn	DementevNikita
Damir Abdullin	LinkedIn	damirabdul
Alexey Kozyurov	LinkedIn	Leshe4ka
Pavel Makarichev	LinkedIn	vixtir
Roman Zabaluev	LinkedIn	Haarolean

License

ODD Platform uses the Apache 2.0 License.