Designed by Agile Lab, witboost is a versatile platform that addresses various sophisticated data engineering challenges. It enables businesses to discover, enhance, and productize their data, fostering the creation of automated data platforms that adhere to the highest standards of data governance. Want to know more about witboost? Check it out here or contact us!

This repository is part of our Starter Kit meant to showcase witboost's integration capabilities and provide a "batteries-included" product.

Tonic AI Adapter

Overview

This project provides integration between Witboost and Tonic.AI to automatically generate a synthetic output port starting from the real one.

Details

Access control is required to protect real output ports. A consumer must request access and wait for approval. Witboost can automate the creation of a fake output port containing synthetic data, and this output port can be left without access management, so it will be super easy for a consumer to get a preview of the data and also start a prototype without waiting for its access granting to the real output port

Building

Requirements:

Python 3.11
Poetry

Installing:

To set up a Python environment we use Poetry:

curl -sSL https://install.python-poetry.org | python3 -

Once Poetry is installed and in your $PATH, you can execute the following:

poetry --version

If you see something like Poetry (version x.x.x), your install is ready to use!

Install the dependencies defined in specific-provisioner/pyproject.toml:

cd specific-provisioner
poetry install

Note: All the following commands are to be run in the Poetry project directory with the virtualenv enabled.

Type check: is handled by mypy:

poetry run mypy src/

Tests: are handled by pytest:

poetry run pytest --cov=src/ tests/. --cov-report=xml

Artifacts & Docker image: the project leverages Poetry for packaging. Build package with:

poetry build

The Docker image can be built with:

docker build .

More details can be found here.

Note: the version for the project is automatically computed using information gathered from Git, using branch name and tags. Unless you are on a release branch 1.2.x or a tag v1.2.3 it will end up being 0.0.0. You can follow this branch/tag convention or update the version computation to match your preferred strategy.

CI/CD: the pipeline is based on GitLab CI as that's what we use internally. It's configured by the .gitlab-ci.yaml file in the root of the repository. You can use that as a starting point for your customizations.

Running

To run the server locally, use:

cd specific-provisioner
source $(poetry env info --path)/bin/activate # only needed if venv is not already enabled
uvicorn src.main:app --host 127.0.0.1 --port 8091

By default, the server binds to port 8091 on localhost. After it's up and running, you can make provisioning requests for this address. You can also check the API documentation served here.

Env Variables

In order to set the application, some environmental variables must be injected:

WorkspaceId: For this prototype, the workspace will be one, but potentially would be better to have one workspace for data product
TonicAI API Key: you can retrieve it from your account setting panel of Tonic.AI

workspaceId = os.getenv('WorkspaceId')
apiKey = os.getenv('API_KEY')

Deploying

This microservice is meant to be deployed to a Kubernetes cluster with the included Helm chart and the scripts that can be found in the helm subdirectory. You can find more details here.

License

This project is available under the Apache License, Version 2.0; see LICENSE for full details.

About us

From Chaos to Control: Data Experience on Guardrails Craft your own standards and automate your data practice

agile-lab-dev/TonicAI-Adapter