Website · Slack Community · Documentation · Blog
valmi.io is the open-source Reverse-ETL (data activation) platform to load data from warehouses into SaaS platforms, Webhook Apis etc.
valmi.io uses some of the best tools to create an Open Source reverse-ETL (data activation) Platform. It is built over the airbyte protocol. dbt is the centerpiece of our source connectors, and duckdb for metrics. We engineered our orchestrator over dagster, and dagster dovetails perfectly with our vision of being a multi-persona tool.
We envision a world where a vibrant community of engineers develops around connectors - a world in which the power of the open-source platform draws on the collective mind to keep the fast-moving world of connectors functional and cost-effective.
-
Checkout docs at - https://docs.valmi.io
-
Read the variety of usecases valmi.io can enable for your organization - https://blog.valmi.io/
-
- You can immediately create a sync at https://demo.valmi.io.
- Watch a demo video to create a sync.
-
Clone this repo and move into the directory.
git clone git@github.com:valmi-io/valmi-activation.git cd valmi-activation git submodule update --init --recursive
-
Setup the environment.
cp .env-example .env cd valmi-app-backend cp .env-example .env cd ../valmi-app `For macos` cp .env-example.macos .env `For linux` cp .env-example.linux .env
-
Intermediate storage, We are adding support for object stores like S3, GCS. Until then, Local storage is used.
sudo mkdir -p /tmp/shared_dir/intermediate_store sudo chmod -R 777 /tmp/shared_dir/intermediate_store
-
Launch the reverse-etl service.
./valmi prod
-
To stop the service, run the following.
./valmi prod down
-
Please wait for about 2 minutes before you access the service, since valmi-app builds an optimized compiled version of the app UI. To access the service, please check the 'Accessing the service' section for local deployments.
3. Develop a connector locally to customize valmi.io as per your needs. You can just contact us too.
-
Clone, setup environment variables and create intermediate storage (see above section).
-
Create a new connector (Optional).
# Copy code base from any existing connectors from valmi-integrations folder (ex. destination-webhook) cd valmi-integrations/connectors cp -r destination-webhook destination-awesome_connector # Make necessary changes and build the connector cd destination-awesome_connector make build_docker version=latest # Add the new connector information to "valmi-app-backend/init_db/connector_def.json"
-
Run the service.
./valmi dev
-
To access the service, please check the 'Accessing the service' section for local deployments.
-
To Stop the service, run the following.
./valmi dev down
-
Syncs
http://localhost:3000Sync Runs
http://localhost:3000UI Backend Server API
http://localhost:4000/api/docsActivation Server API
http://localhost:8000/docs