Being able to `tail` a topic would be a great feature

[krisajenkins]

One tool I use all the time for debugging is kcat -b <broker_address> -Ct <topic_name>. That essentially does tail -f <topic_name> - it lets me watch the end of the topic in real time to see what data's coming in. The quix CLI current makes this harder, but in a way I think reveals a couple of useful new features.

So imagine I'm running a pipeline with quix pipeline up and I want to watch a topic. How do I do it? How do I find the broker address and the topic name?

Where's the Broker Address?

It's probably localhost:19092. But I think quix pipeline status should tell me explicitly.

What's the topic name?

This is findable - it's going to be in the code, or in app.yaml, or quix.yaml. But that's a lot to look through. quix topic list or quix pipeline topics list would be very useful.

Do we even need kcat?

kcat -Ct is just a consumer. How about we add a basic consumer into the CLI, so you can call quix topic tail <topic_name>? That removes the need for another tool and makes quix CLI even more useful. (Actually, between you and me, kcat is not the most user-friendly tool in the world. I think we could leapfrog ahead of it. 😉)

One more thought - add it to the pipeline view

quix pipeline view generates a nice mermaid diagram from a markdown document. How about we add quix topic list's output as a markdown table?

For reference

Here's another useful kcat command we can take inspiration from. List topics:

$ kcat -b localhost:19092 -L
Metadata for all topics (from broker 0: localhost:19092/0):
 1 brokers:
  broker 0 at localhost:19092 (controller)
 5 topics:
  topic "_schemas" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "__consumer_offsets" with 3 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 0, replicas: 0, isrs: 0
    partition 2, leader 0, replicas: 0, isrs: 0
  topic "weather_data_demo" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "weather_i18n" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "changelog__weather_to_google--weather_data_demo--tumbling_window_3600000_reduce" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0

Related: http://blog.jenkster.com/2022/10/setting-up-kcat-config.html

To Summarize

I think we should add:

quix pipeline status          # Now shows the broker address.
quix pipeline view            # Now shows topics and their metadata as well.
quix topic list
quix topic tail <topic_name>

[stereosky]

Thank you! I like this suggestion. For bonus points add retention time and size 🤯