synthetic-producer

Generates random JSON messages and sends them to Apache Kafka. It uses Apache Zookeeper to discover the available brokers and partitions for a given topic.

Usage:

$ java -jar target/synthetic-producer-1.4.0-selfcontained.jar
usage: java -jar synthetic-producer.jar
 -c,--config <arg>      config file path
 -h,--help              show this help
 -r,--rate <arg>        messages rate per second
 -t,--threads <arg>     number of producer threads
 -z,--zookeeper <arg>   zookeeper connect string

Config file

The config file specifies the Zookeeper connection string, the rate of messages generated, the threads to use, the topic where you want to produce, and the messages' schema.

For example, take a look at the following config:

topic: testTopic
partitionKey: myPartitionKeyField
fields:
  myConstant:
    type: constant
    value: 'Hello, World!'
  randomNumber:
    type: integer
    min: 0
    max: 100

Using a file like that will make the producer will connect to Zookeeper at "localhost:2181" in order create two producing threads, and send messages to the topic "testTopic" at a constant rate of 100 msgs per second, which would look like the following:

{ "myConstant": "Hello, World!", "randomNumber": 4}
{ "myConstant": "Hello, World!", "randomNumber": 12}
{ "myConstant": "Hello, World!", "randomNumber": 94}
{ "myConstant": "Hello, World!", "randomNumber": 33}

You can find a full example of config file in this file

Fields types

You can use a variety of different field types to generate your random message. Some types accepts one or more parameters that modify the field randomness. For example, the constant type always generates the same value for a field, but the integer type generates a random number between 'min' and 'max' for each field.

The available types, with their parameters, are the following:

constant

Generates a constant value

Parameters:

value: The value to be generated

Example:

myField:
 type: constant
 value: "This is a constant"

integer

Generates a random integer

Parameters:

min: The minimum integer to be generated (Default 0)
max: The maximum integer to be generated (Default Integer.MAX_VALUE)
negative: True if the generated number must be negative (It will be multiplied by -1)

Example:

myField:
  type: integer
  min: 0
  max: 50

double

Generates a random double

Parameters:

min: The minimum double to be generated (Default 0.00)
max: The maximum double to be generated (Default Double.MAX_VALUE)
negative: True if the generated number must be negative (It will be multiplied by -1)
truncate: The decimals numbers (Default 10000, four decimals)

Example:

myField:
  type: double
  min: 0.00
  max: 50.00

counter

Generates an integer counter between two values

Parameters:

min: Minimum value that counter can return (Default 0)
max: Maximum value that counter can return (Default Integer.MAX_VALUE)
direction: True if the counter goes from minimum to maximum. False if it goes from maximum to minimum. (Default true)

Example:

myField:
  type: counter
  min: 1
  max: 10

ip

Generates a random IP address

Parameters:

min: The minimum ip to be generated (Default 0.0.0.0)
max: The maximum ip to be generated (Default 255.255.255.255)

Example:

myField:
  type: ip
  min: 192.168.0.1
  max: 192.168.0.255

mac

Generates a random MAC address

Parameters:

min: The minimum ip to be generated (Default 00:00:00:00:00:00)
max: The maximum ip to be generated (Default FF:FF:FF:FF:FF:FF)
separator: The string using to split the mac on the output.

Example:

myField:
  type: mac
  min: 00:00:00:00:00:11
  max: 00:00:00:00:00:44

timestamp

Generates a field with the current timestamp of the machine

Example:

myField:
  type: timestamp

collection

Generates a random element from the given collection of elements

Parameters:

values: An array of elements

Example:

myField:
  type: collection
  values:
    - Campus A
    - Campus B
    - Campus C
    - Campus D
    - Campus E

composition

Generates a random field based on a composition of two or more random fields

Parameters:

separator: The character that will be used as the separator between the two generated values
components: An array of types definitions that will be generated before joining the result with the separator

Examples:

myField:
  type: composition
  separator: ':'
  components:
    - type: integer
      min: 0
      max: 11
    - type: integer
      min: 0
      max: 101

This will create values like the following: 2:34, 6:11, 1:75, 10:62... Of course, you can use a composition type as a component for another composition type.

set

Generate a fixed number of sets with multiple fixed field per set.

Parameters:

numbers: The different numbers of sets
components: The components that form a set

  set123:
    type: set
    numbers: 10
    components:
      client_mac:
          type: mac
          min: 00:00:00:00:00:11
          max: 00:00:00:00:00:44

      src:
          type: ip
          min: 80.82.34.12
          max: 80.82.34.72

This will create 10 sets form by a client_mac of type mac and src of type ip. The sets are making on the init and them don't change after. A message has 1 set that is choose randomly from 10 sets.

random

Choose one component randomly among all the components inside it.

Parameters:

components: List of the components that will be picked randomly.

You can find a simple example below:

  mySimpleRandom:
    type: random
    components:
      -
        myField:
          type: integer
          min: 101
          max: 110
      -
        myField:
          type: integer
          min: 201
          max: 210
      -
        myField:
          type: integer
          min: 301
          max: 310

This will generate values from 101 to 110, from 201 to 210 and from 301 to 310. Also you can nest random types in order to get more complex combinations.

json

Generate a new json inside other json.

Parameters:

components: The components that form a set

  engine_id_name:
    type: constant
    value: IANA-L4
  myNewJson:
    type: json
    components:
        src:
          type: ip
          min: 192.168.0.1
          max: 192.168.255.255
        application_id:
          type: composition
          separator: ':'
          components:
             - type: integer
               min: 0
               max: 11
             - type: integer
               min: 0
               max: 101
        myNewJsonInside:
          type: json
          components:
             dst:
               type: ip
               min: 192.168.0.1
               max: 192.168.255.255

This will generate messages like this:

{"myNewJson":{"myNewJsonInside":{"dst":"192.168.231.69"},"src":"192.168.161.220","application_id":"9:32"},"engine_id_name":"IANA-L4"}

coordinates

Generate a random latitude and longitude inside a circle that is defined using radius.

latPoint: The latitude center point.
longPoint: The longitude center point.
radius: The radius of the circle.
latitudeDim: The field name of the latitude value.
longitudeDim: The field name of the longitude value.
truncate: The decimals numbers (Default 10000, four decimals)

Example:

  coordinates:
    type: coordinate
    latPoint: 30.000
    longPoint: -40.000
    radius: 1000
    latitudeDim: latitude
    longitudeDim: longitude
    truncate: 1000

array

Generate array with different components.

components: The different types that build the array

Example:

myArray:
    type: array
    components:
       field1:
         type: json
         components:
            dst:
               type: ip
               min: 192.168.0.1
               max: 192.168.255.255
       field2:
         type: constant
         value: 5

This example generate an array like:

{"myArray":[5,{"dst":"192.168.142.39"}]}
{"myArray":[5,{"dst":"192.168.121.126"}]}
{"myArray":[5,{"dst":"192.168.189.54"}]}

sequential

Generate one component each time. The sequential restart when reach the end.

components: The different types that build the sequence

sequentialField:
 type: sequential
 components:
   -
     seq1:
       type: set
       numbers: 1
       components:
         longitude:
            type: constant
            value: 1
         latitude:
            type: constant
            value: -1
   -
     seq2:
       type: set
       numbers: 1
       components:
         longitude:
            type: constant
            value: 2
         latitude:
            type: constant
            value: -2
   -
     seq3:
       type: set
       numbers: 1
       components:
         longitude:
            type: constant
            value: 3
         latitude:
            type: constant
            value: -3

This example generate an array like:

{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}
{"longitude":3,"latitude":-3}
{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}

Contributing

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

License

AGPL v3

redBorder/synthetic-producer

synthetic-producer

Config file

Fields types

constant

integer

double

counter

ip

mac

timestamp

collection

composition

set

random

json

coordinates

array

sequential

Contributing

License