vvalisoy/valmi-activation

valmi.io reverse-ETL (data activation) is the open-source data activation platform to load data from warehouses into SaaS platforms, Webhook Apis etc.

Website · Slack Community · Documentation · Blog

valmi.io is the open-source Reverse-ETL (data activation) platform to load data from warehouses into SaaS platforms, Webhook Apis etc.

• See it in Action

  Quickly witness a One Million (1M) Rows Sync at Live Sync.

  
  

valmi.io uses some of the best tools to create an Open Source reverse-ETL (data activation) Platform. It is built over the airbyte protocol. dbt is the centerpiece of our source connectors, and duckdb for metrics. We engineered our orchestrator over dagster, and dagster dovetails perfectly with our vision of being a multi-persona tool.

We envision a world where a vibrant community of engineers develops around connectors - a world in which the power of the open-source platform draws on the collective mind to keep the fast-moving world of connectors functional and cost-effective.

• Checkout docs at - https://docs.valmi.io

• Read the variety of usecases valmi.io can enable for your organization - https://blog.valmi.io/

• 3 ways to start with valmi.io

  1. Use it in the cloud

  • You can immediately create a sync at https://demo.valmi.io.
  • Watch a demo video to create a sync.
  
  

  

  2. Run it locally or in your Cloud

  • Clone this repo and move into the directory.

    git clone git@github.com:valmi-io/valmi-activation.git
    cd valmi-activation
    git submodule update --init --recursive

  • Setup the environment.

    cp .env-example .env
    
    cd valmi-app-backend
    cp .env-example .env
    
    cd ../valmi-app
    `For macos`
    cp .env-example.macos .env
    `For linux`
    cp .env-example.linux .env

  • Intermediate storage, We are adding support for object stores like S3, GCS. Until then, Local storage is used.

    sudo mkdir -p /tmp/shared_dir/intermediate_store
    sudo chmod -R 777 /tmp/shared_dir/intermediate_store

  • Launch the reverse-etl service.

    ./valmi prod

  • To stop the service, run the following.

    ./valmi prod down

  • Please wait for about 2 minutes before you access the service, since valmi-app builds an optimized compiled version of the app UI. To access the service, please check the 'Accessing the service' section for local deployments.

  3. Develop a connector locally to customize valmi.io as per your needs. You can just contact us too.

  • Clone, setup environment variables and create intermediate storage (see above section).

  • Create a new connector (Optional).

    # Copy code base from any existing connectors from valmi-integrations folder (ex. destination-webhook)
    
    cd valmi-integrations/connectors
    cp -r destination-webhook destination-awesome_connector
    
    # Make necessary changes and build the connector
    cd destination-awesome_connector
    make build_docker version=latest
    
    # Add the new connector information to "valmi-app-backend/init_db/connector_def.json"

  • Run the service.

    ./valmi dev

  • To access the service, please check the 'Accessing the service' section for local deployments.

  • To Stop the service, run the following.

    ./valmi dev down

• Accessing the service for local deployments

  Syncs http://localhost:3000	Sync Runs http://localhost:3000

  UI Backend Server API http://localhost:4000/api/docs	Activation Server API http://localhost:8000/docs