Kuper-Tech/protokaf

Kafka producer and consumer tool in protobuf format.

protokaf

Kafka producer and consumer tool in protobuf format.

Features

• Consume and produce messages using Protobuf protocol
• Trace messages with Jaeger
• Create custom templates for one or multiple messages and produce them to Kafka

Install

go install github.com/kuper-tech/protokaf@latest

Configuration

Configuration file is optional, so you can skip this section.

In order for Protokaf to work, it needs to know how to reach your Kafka broker. First option is to provide --broker each time you invoke Protokaf. Another option is to use a configuration file. You can provide Protokaf with a configuration file with option -F .. on the command line. Or by default Protokaf will search its config files in .protokaf.yaml and $HOME/.protokaf.yaml respectively.

Example of .protokaf.yaml

debug: true
broker: "<addr>:<port>"
kafka-auth-dsn: "SCRAM-SHA-512:<namespace>:<passwd>"
proto: "<dir>/<protofile>"

Help

$ protokaf help

List metadata

$ protokaf list [-t <topic>(,<topic>...)]
1 brokers:
 broker 1 "127.0.0.1:9093"
2 topics:
  topic "test-topic", partitions: 1
    partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]
  topic "test", partitions: 1
    partition 0, leader 1, replicas: [1] (offline: []), isrs: [1]

Produce

Help

$ protokaf produce -h

Examples

This proto file will be used in the examples below.

api/example.proto

syntax = "proto3";

package example;

message HelloRequest {
  string name = 1;
  int32 age = 2;
}

A simple produce message

$ protokaf produce HelloRequest \
    --broker kafka:9092 \
    --proto internal/proto/testdata/example.proto \
    --topic test \
    --data '{"name": "Alice", "age": 11}'

Produce message with headers

$ protokaf produce HelloRequest \
    --broker kafka:9092 \
    --proto internal/proto/testdata/example.proto \
    --topic test \
    --header "priority=high" \
    --header "application=protokaf" \
    --data '{"name": "Alice", "age": 11}'

Produce message with template

$ protokaf produce HelloRequest \
    --broker kafka:9092 \
    --proto internal/proto/testdata/example.proto \
    --topic test \
    --data '{"name": {{randomFemaleName | quote}}, "age": {{randomNumber 10 20}}}' \
    --count 10 \
    --seed 42

Produce message with Kafka auth

$ protokaf produce HelloRequest \
    --broker kafka:9093 \
    --kafka-auth-dsn "SCRAM-SHA-512:login:passwd" \
    --proto internal/proto/testdata/example.proto \
    --topic test \
    --data '{"name": "Alice", "age": 11}'

Read data from stdin or flag

Read message HelloRequest from stdin, produce to test topic

$ echo '{"name": "Alice", "age": 11}' | protokaf produce HelloRequest -t test

Read message HelloRequest from -d value, produce to test topic

$ protokaf produce HelloRequest -t test -d '{"name": "Alice", "age": 11}'

Template

Template options

• --seed <int> You can set number greater then zero to produce the same pseudo-random sequence of messages
• --count <int> Useful for generating messages with random data
• --concurrency <int> Number of message senders to run concurrently for const concurrency producing

Show all template functions

$ protokaf produce --template-functions-print

Build json template by proto file

This can be useful for creating body for produce command

$ protokaf build HelloRequest --proto internal/proto/testdata/example.proto

For proto file

syntax = "proto3";

package example;

message HelloRequest {
  enum Status {
    PENDING = 0;
    COMPLETED = 1;
  }

  string name = 1;
  int32 age = 2;
  optional float amount = 4;
  Status status = 5;
  repeated string keys = 6;
}

command will print

{
  "name": "",
  "age": 0,
  "amount": 0,
  "status": "PENDING",
  "keys": [
    ""
  ]
}

Consume

Help

$ protokaf help consume

Examples

$ protokaf consume HelloRequest \
    --broker kafka:9092 \
    --proto internal/proto/testdata/example.proto \
    --group mygroup \
    --topic test

Read messages from Kafka test topic, use group mygroup, print to stdout

$ protokaf consume HelloRequest -G mygroup -t test

Read the last 10 messages from test topic, then exit

$ protokaf consume HelloRequest -G mygroup -t test -c 10

Read from offset 5 messages from test topic

$ protokaf consume HelloRequest -G mygroup -t test -o 5

Testing

Prepare test environment and running tests

make docker-dev-up
make kafka-users
make test
make install # optional (you can use 'go run . <args> <flags>')