Custom Generator
povils opened this issue · 12 comments
Hey team!
What is the best way to write your own custom generator if needed? Like if I want to retrieve all clusters from the certain REST API, it would be great just to plug in my own built custom generator
I like this idea a lot.
One way would be to write a REST generator and then parameterize the heck out of it. But that seems fragile.
Argo Workflows has a plugin system which relies on the plugin author providing a sidecar which implements a well-defined HTTP server. The server can be written in any language and basically do anything the author wants, as long as it responds to the right endpoints.
But maybe that's more complicated than what you need. What do you think?
Hey! I think it would work and would be quite flexible :) The only concern is wouldn't we still need to parametrize whatever(I assume AppliationSet controller) calls those well-defined endpoints? Like:
/v1/clusters?region=eu&environment=test
With this in mind wouldn't just be easier to have HTTP generator and you can set HTTP headers and body whatever you like? Of course, it will only then solves HTTP problem... What if you want to get clusters from PostgresDB ?:D
I think HTTP server as a sidecar is the most flexible option. We just have to make sure that this http-sidecar-custom-gernerator is flexible enough to make clusters filterable
Makes sense! I think we could make a JSON body part of the contract and then pass a generic object as one of the fields for parameters. That should leave plenty of flexibility for folks to send the plugin whatever information it needs.
@crenshaw-dev what needs to be worked on? Do you have some design in mind already?
@povils I think we have enough of a rough sketch for a design that a proof of concept could be built. Unfortunately I won't have time in the foreseeable future to prioritize this. If you have some time, I'd be happy to help you through any dev setup issues.
Could you share the sketch with me, I could contribute :)
I think you'd just define a new HTTPGenerator which accepts url as a parameter. Then accept as a response any JSON list of objects. The key/value pairs in the objects would become the parameters. There's already logic in the code which could be borrowed to flatten the JSON list.
@crenshaw-dev Will only URL be enough? What if you want some kind of authentication (Bearer token header) or provide a custom CA for TLS, and etc.
We can start with only URL and go from there ofc :)
If we limit it to localhost for sidecar plugins at first, I think we can skip auth. But yes, eventually we'd need to be able to parameterize it with secret references to populate auth headers.
that sounds like a good start :)
Guys, i think this idea is super cool. I've started to use application sets and realized that for source of truth i want to use some external database, for example DynamoDB. This is really helpful for dynamic env spin ups, like playgrounds....
Some generator which gets data from somewhere by rest it's fantastic. If you need help for contribution you can rely on me
I came across this suggestion a while back when I was about to post same request and forgot to give it a thumb up. We have tons of ideas where a custom generator is needed, quite often involving topology which is always quite specific.
It s true it might be perceived as not Gitops (when using db as source etc... ) But like it a lot