Async communication
Closed this issue · 17 comments
i really like the approach this framework takes of using annotations.
i would like to try to add async communication generators .
i use NATS quite a bit . It’s a message bus and broker .
https://github.com/nats-io/nats.go
It has HTTP / web socket and tcp transports built in.
So for iot people use the tcp layer .
But for web they use the http and web sockets .
Nats is globally clustered , self load balancing and has build in security. It also has a clustered kv that replaced etc and consul.
So it remove the need for a lot of other layers typically needed for global internet scale systems engineering .
It’s 100% golang and runs on anything .
It can do RPC and PUB SUB also.
You can use protobufs, json , anything you want serialisation . It does not care .
I don’t know how familiar you are with NATS . But just let me know if this is something if interest.
i currently use it in iot, web, desktop and servers .
—-
there is a standard approach to async communications that is emerging also called “async api”.
It’s like openapi but for async.
it also has generators but like openapi takes the approach of you first writing your api description in json or yaml and then code generating from that .
it’s quite possible however to generate an async api document using kok !!!! This I think would be pretty cool because then others can generate the client in whatever language they want .
https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=
Hi @gedw99, thank you for the useful information!
I do know NATS, but have little experience with it. As for the asynchronous specification, I've used CloudEvents, but have no idea about AsyncAPI, which is awesome!
Your idea of generating AsyncAPI document sounds interesting. Is there any example in your mind? What does the input and output of the generator look like?
Well cloud event is agnostic and uses nats if you want.
there are no examples of asyncapi, but its not really needed because the core part is all golang and really well done from what i can see
https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=
https://github.com/asyncapi/event-gateway currently uses Kafka, but i think adding nats is not hard actually.
NATS is such a simple api. Their go.mod shows it uses watermill, the nats watermill is already there that can be integrated: https://github.com/ThreeDotsLabs/watermill-nats
Also from the go.mod you can see the gateway uses this parser github.com/asyncapi/parser-go, which i think is parsing the schema spec.
https://github.com/asyncapi/spec-json-schemas holds the spec versions. Pretty simple.
https://github.com/asyncapi/converter-go is just a spec upgrader.
The way this golang stuff hangs together is pretty clean IMHO.
All the other repos in async api is all just for JS developers.
Its mostly Tons of complex tooling so as to give them a web based GUI.
Not needed in many ways. But there is a developer wants it.
i started a discussion here: asyncapi-archived-repos/event-gateway#81
Also the gateway can then be part of the Caddy proxy if you wanted. Just saw that you have one.
So then all Sync and async operations can be described from the Spec and used by caddy to forward everything to the correct Service.
Could also of course be just run inline.
Your way of using annotations means that the spec is generated
Its kind of cool because you can embed the caddway and NATS Jetstream into a simple binary IF you want OR you can scale it out. Cant do that with kafka.
ah i just realised that there are bindings.
Its a late bound architecture...
https://github.com/asyncapi/bindings/blob/master/nats/README.md
Found an example: https://github.com/asyncapi/ts-nats-template
they lean towards generating JS based clients.
but generating nats or other golang client is possible of course.
Sounds great! Welcome to add the AsyncAPI document generator you expected ;)
Just FYI, here are the generation methods currently used in kok:
- Runtime reflection (based on reflect) for generating OpenAPI Specification
- We can not use static analysis here, since the requests and responses may be manipulated dynamically by custom HTTP codecs.
- Static analysis (based on go/packages) for generating Protocol Buffers
sorry about not following up.. Got snowed under with other work stuff.
I realised after messing around with ideas that the most powerful thing that i can do with kun is to parse golang structs to JSONSchema. Then at runtime i can use these JSON Schemas for runtime checks and events and a myriad of things.
Its nice too because you work in Golang, gen your JSONSchema that represent those structs. Forward engineering.
Then you can build systems that are loosely coupled at runtime.
Hi @gedw99, here is an attempt to add support for generating handlers for Event Subscribers #27. (Does not yet support generating AsyncAPI documentation.)
I realised after messing around with ideas that the most powerful thing that i can do with kun is to parse golang structs to JSONSchema.
Sounds good! Then it's recommended to use static analysis (based on go/packages). Maybe you could take a look at the implementation of the gRPC parser :)
Hey Hey
Looks great. Whats the transport ? I assume websockets ? I ask because i want to be able to compile the generated golang client so that the GUI works with GIOUO: https://github.com/gioui. nhooyr.io/websocket works with GIOUI.
here is a graphql client that uses it and works with GIOUI just to give you an example of what i mean: https://github.com/hasura/go-graphql-client
About AsyncAPI. You want to use their system to run it ?
Whats the transport ? I assume websockets ?
The current design is meant to be generic, so there's no assumption about the transport for now. We currently use Kafka, but any other transport can use the generated handlers.
About AsyncAPI. You want to use their system to run it ?
Not intend to do so. But I think it's specially suitable for generating documentation (much like OpenAPI).
I don’t us Kafka . Is it possible you could add a Kafka docker to the repo to get me going ? As well as a make file to make it obvious how to use it
As mentioned above, any transport/broker (e.g. WebSocket, Kafka, Redis or even HTTP) can use the generated handlers. As an example, eventsvc just use a simple Go program to trigger the handlers.
thanks the eventsrc example helps..