/AvroProducer

A Simple Python based Avro Kafka producer to create records from a CSV file and an AVRO Schema

Primary LanguagePython

Purpose

Sometimes during building data piplines on Kafka , there is a need to

  • Quickly build test data for data stream testing
  • Provide the AWESOMENESS of AVRO to create data records constrained to a Schema(AVRO)

Initial Setup

Install self-contained binary for OSX and Linux:

   apt install confluent-kafka

Install AvroProducer and AvroConsumer:

   pip install confluent-kafka[avro]

Usage

  • Replace the *.avsc file with your schema
  • Replace the *.csv file with yours
  • Run You kafka cluster with the supplied docker-compose.yml docker-compose up
  • Run the producer python produce.py