Apache Kafka Streams + Spring Boot App
This is an example Spring Boot app that uses Apache Kafka to read payment events from one Kafka topic, adds fees to the payment and writes it back to a new topic.
The project contains 3 modules
Consumer
Apache Kafka Streams + Spring Boot application that reads from a payments
(configurable) topic, adds fees to the payment
object and writes it back to updated_payments
topic.
Producer
Produces random Payment
events that is to be used for testing.
Schemas
This module contains Payment
and Currency
Avro IDL files that is used when both consuming and producing payments.
How to Run
- Create a new Confluent Cloud account create a new cluster
- Generate API keys for both Kafka API and Schema Registry API
- Add the following authorisation to the generated Kafka API access keys
- Cluster:
kafka-cluster
-ALLOW
-IDEMPOTENT_WRITE
- Consumer Group ID:
confluent-payment-stream
-ALLOW
-READ
- Topic:
*
-ALLOW
-READ
- This gives read access to all topics in the cluster (You should not do this in a production environment) - Topic:
*
-ALLOW
-WRITE
- This gives write access to all topics in the cluster (You should not do this in a production environment)
- Cluster:
- Create two topics in Confluent named
payments
andupdated_payments
- Set the following environment variables in your shell OR if you are running through IntelliJ, in the launch configuration
export BOOTSTRAP_SERVERS="$bootstrapServers"
export SCHEMA_REGISTRY_URL="$schemaRegistryUrl"
export BASIC_AUTH_USER_INFO="$SchemaRegistryApiKey:$schemaRegistryApiSecret"
export SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$kafkaApiKey\" password=\"$kafkaApiSecret\";"
- Run
./gradlew payment-event-producer:run
to by default produce 5 payment records. You can pass--args=$numberOfEvents
to generate any number of payment events. E.g./gradlew payment-event-producer:run --args="10"
- Run
./gradlew payment-event-consumer:run
to start the consumer. The consumer will print the payment event before and after adding fees.