GPTScript is great to glue together various system (admin) utilities and DevOps tools, just as what shell scripts did, although written in human languages.
chat-kafka is a thin wrapper over kaf, provides a set of tools and a vocabulary of words and sentences for gptscript to "talk" to kafka broker cluster, such as "connect to broker at localhost:9092", "create a topic A", "send message to topic A", etc. Check out examples/data-pipeline.gpt for more examples, which is a demo data pipeline: start local broker in docker, create topics, ingest log data, data transformation and aggregation.
Also, unix style pipes are supported over kafka topics (experimentally), such as "use topics to pipe ls to sort reverse to translate lower to upper case". Pipes also simplify data-pipeline.
To use this toolset/package in your gptscript, add "github.com/yglcode/chat-kafka" in the "tools: " section.
To run the example, you need:
- install gptscript and set up OpenAI api key as instructed in gptscript project.
- have docker installed.
- clone this repo
- cd examples
- gptscript --cache=false data-pipeline.gpt
- gptscript --cache=false pipe_thru_kafka.gpt