The purpose of this project is to track the specification for the Kobold General Purpose Messaging Protocol (KGPMP) throughout it's development. This spec will describe the transfer of data between nodes
and clients
. the KGPMP is meant for the the Kobold Distributed Messaging System (KDMS) under active development. Kobold is an in memory messaging system that uses a client/server model where messages are sent from a client
to a node
where they are then routed to other clients
.
kgpmp
is meant to support the following functionalities to facilitation communication between systems. These systems can be running on a single machine on an isolated network or a cluster of machines working together across the world.
- unidirectional
client
->node
- bidirectional
client
<->node
- unidirectional
node
->node
- bidirectional
node
<->node
The purpose of KGPMP is to standardize message passing between clients
and nodes
. The protocol is meant to simplify and reduce overhead. It is meant to be easily parsed and turned into useable data structures within most modern programming languages. There are common patterns that keep coming up over and over again that can be solved simply and efficiently when communicating between distributed systems across multiple languages.
Core patterns
Pattern | Description |
---|---|
Request/Reply | a client sends a transaction to be acknowledged and returned by another client |
Publish/Subscribe | n message producers sending messages to x consumers without a buffer |
Non Core patterns
Pattern | Description |
---|---|
Message Queue | n producers enqueue messages and x consumers dequeue messages at their own pace |
Data Store | ability to CRUD a key/value from multiple clients |
Stream | send a stream of bytes over a persistent connection |
Messages that are sent from clients to nodes
Message Type | Description |
---|---|
Publish | (push) publish a message to a topic |
Request | send a request to a service topic. |
Reply | send a reply to a request. |
Request Type | Description |
---|---|
Forward | forward the request to a client advertised service |
Authenticate | authenticate the connection |
Subscribe | subscribe to a topic |
Unsubscribe | unsubscribe from a topic |
Advertise | advertise a service topic |
Undvertise | unadvertise a service topic |
NodeInfo | get information about a node |
Topic Keywords | Description |
---|---|
$node | node the client is currently connected to |
type Publish<T> struct {
id: string
topic: string
metadata: Metadata // information about the client/connection
message: T
}
---
// client creates a request
// client dispatches requests to node w/ tx_id
// node receives request
// node enqueues request in service request queue
// service dequeues next request
// service handles request
// service dispatches reply to node w/ tx_id
// node enqueues reply in client reply queue
// client handles reply
type Request<T> struct {
id: string
request_type: RequestType,
topic: string
tx_id: string
metadata: Metadata // information about the client/connection
message: T
}
---
type Reply<T> struct {
id: string
topic: string
tx_id: string
metadata: Metadata // information about the client/connection
message: T
}
A message must be able to pass from a client -> node -> n
nodes -> x
clients
. So forwarding a message while maintaining it's integrity is essential. So in order to support larger payloads, we will have to cut messages into chunks. The simplest way I can think to do this is to add a 2 fields to the metadata struct. part
, total_parts
. The receiving client can simply track all parts of the message with id
and reconstruct the larger message.
Client based handlers
So very simply, the receiving client can receive any chunk of a message and know the total number of parts of the message. If a part was not received in the correct amount of time, the client could ask for a retransmission of just that part. The receiver client would just need to retrace the message back to it's origin and request that specific part. Although I think the functionality of this is necessary, I think it is quite cumbersome and could be done better.
Node based handlers
Another easy way to get data verification is for clients to upload the entire message to the node
and have the node validate that all parts are received. This puts the data validation part on the node
and removes the complexity of retracing a message back to the originating client. It does put more demand on available disk or memory of the node.
term | definition |
---|---|
prefix | 4 byte integer describing the number of bytes included in this message |
proto | an encoded struct containing the required fields to route and handle the message |
Each message is to be prefixed by a 4 byte integer describing the length of the message. This includes all delimiters included.
<prefix><proto>
Examples
# request
<prefix><encoded message>
TLDR; cbor
seems to be the best starting point for encoding that I can come up with.
although my tests are shitty and primitive, they show that json is slow as fuck and that Cap'n Proto and CBOR are pretty similar. There are great benefits to using either.
This is all running on my local machine and therefore all tests are uncontrolled. This is all biased, I know. I'm not interested in being really thorough. There are some obvious optimizations that can happen. I'm sure I could write a faster or more efficient encoding system but I doubt it will be as robust as any of these others. cbor
wins for now.
Encoding | Iterations | Serialization(ms) | Parsing(ms) | Deserialization(ms) | Total Time(ms) |
---|---|---|---|---|---|
Cap'n Proto | 1_000_000 | 1250.15 | 96.87 | 334.78 | 1681.90 |
CBOR | 1_000_000 | 725.97 | 86.38 | 672.57 | 1484.96 |
Msgpack | 1_000_000 | untested | untested | untested | untested |
JSON | 1_000_000 | 1095.81 | 122.90 | 1595.00 | 2813.74 |
goos: linux
goarch: amd64
pkg: github.com/bahodge/kgpmp-prototype
cpu: 13th Gen Intel(R) Core(TM) i9-13900K
BenchmarkRunCapn1
BenchmarkRunCapn1-24 2774 418643 ns/op
BenchmarkRunCapn100
BenchmarkRunCapn100-24 2532 514270 ns/op
BenchmarkRunCapn10000
BenchmarkRunCapn10000-24 69 17829592 ns/op
BenchmarkRunCapn100000
BenchmarkRunCapn100000-24 6 169682814 ns/op
BenchmarkRunCapn1000000
BenchmarkRunCapn1000000-24 1 1751991028 ns/op
BenchmarkRunCBOR1
BenchmarkRunCBOR1-24 3051 373979 ns/op
BenchmarkRunCBOR100
BenchmarkRunCBOR100-24 1687 598904 ns/op
BenchmarkRunCBOR10000
BenchmarkRunCBOR10000-24 72 16247588 ns/op
BenchmarkRunCBOR100000
BenchmarkRunCBOR100000-24 6 177246694 ns/op
BenchmarkRunCBOR1000000
BenchmarkRunCBOR1000000-24 1 1685516807 ns/op
BenchmarkRunMsgpack1
BenchmarkRunMsgpack1-24 4377 401017 ns/op
BenchmarkRunMsgpack100
BenchmarkRunMsgpack100-24 1807 676553 ns/op
BenchmarkRunMsgpack10000
BenchmarkRunMsgpack10000-24 63 18400306 ns/op
BenchmarkRunMsgpack100000
BenchmarkRunMsgpack100000-24 7 179656233 ns/op
BenchmarkRunMsgpack1000000
BenchmarkRunMsgpack1000000-24 1 1929324637 ns/op
BenchmarkRunJSON1
BenchmarkRunJSON1-24 3318 457479 ns/op
BenchmarkRunJSON100
BenchmarkRunJSON100-24 1335 763268 ns/op
BenchmarkRunJSON10000
BenchmarkRunJSON10000-24 44 26974104 ns/op
BenchmarkRunJSON100000
BenchmarkRunJSON100000-24 5 235920948 ns/op
BenchmarkRunJSON1000000
BenchmarkRunJSON1000000-24 1 2910120434 ns/op