pcasaretto/go-block-web-kafka

POC to test blocking web requests on a kafka topic response

Project Description

This is a POC Go program to test some ideas inspired by

• Designing Event Driven Systems by Ben Stopford
• A pratical talk on the subject by Neil Buesing

Usage

1. build the go web server and run a couple of instances of each.
2. send them requests, and watch the logs. you will get a uuid for each
3. use a tool like kcat to "respond" to requests

Useful snippets from my shell history

Using kcat to respond:

echo "hello there" | kcat -v -b localhost:9092 -P -t responses -H "request-id=6c50571c-1224-4e2f-baff-ee077c984158"

Using kcat to monitor requests topic:

kcat \
      -b localhost:9092 \
      -t requests-0 -C \
      -f '\nKey (%K bytes): %k
  Value (%S bytes): %s
  Timestamp: %T
  Partition: %p
  Offset: %o
  Headers: %h\n'

Conclusion

The POC works. I haven't benchmarked it but I think I can handle a lot a troughput. Ideally, we would scale the app vertically as much as we can because having multiple copies of the app runnings means each instance will be processing messages meant for other instances.

Ideas for improvement

• To alleviate allocation on the blocker we could use a sync.Pool