MaterializeInc/datagen

Feature: Add integration tests

bobbyiliev opened this issue · 2 comments

Is your feature request related to a problem? Please describe.

At the moment we are only testing the schema parsing and the output.

It will be nice to actually test the Kafka producer including both JSON and Avro formats

Describe the solution you'd like

We can create a docker-compose file with Kafka + CSR and add additional tests to actually run the datagen script out of dry run mode

Additional context

The tests should run on every PR

We can also add a deploy step to build and upload the docker image to docker hub as well. I’m not sure how deploying to npm works

I'll have to look into this on what permissions would be needed for our Docker Hub account!