How to broadcast PURGE/BAN request to all pods in cluster?
Closed this issue · 14 comments
Hello,
As I'm exploring this operator (good work!) - I found next possible issue. Is it possible to broadcast some request calls among all varnish pods?
Use case would be call all varnish pods and make a cache-tags based cache flush. I found, that one possible solution is to use some sort of HTTP broadcaster as part of varnish containers. I see two alternative solutions:
- https://github.com/mittwald/kube-httpcache#using-built-in-signaller-component
- https://github.com/jderusse/http-broadcast
Is it something that is part of of roadmap for project? Are there alternative solution that may be used with this project?
This is an interesting problem. My initial reaction is that it's outside of the scope of the operator in that it's not a feature the operator automates or controls. Maybe if we added a utility plugin or something like we've done w/our Redis and C* operators, we could support something like this natively. I'm just not sure how many users would find this useful.
I wrote a "kubectl parallel execution" (or kpe) script that runs kubectl commands across pods that match a label in parallel. This allows me to run things like varnishstat or varnishadm across all varnish pods in parallel (i.e. kpe -s -c varnish 'varnishstat | grep SMA'). So a curl to localhost can probably be called w/in varnish pods to do what you want.
Are there any other use cases like this where we'd want to support more interactive management of a live varnish cluster?
Hi, thanks for answer. I believe parallel command execution will not work here (I simply want to run a BAN call from PHP application and make sure that all pods being part of varnish cluster flushed the cache (based on "Cache-Tags" header). I'd rather not give PHP app possibility to use kubectl and run commands directly within kubernetes cluster. So some "magic" replication of HTTP call would be something perfect (I need to investigate how exactly kube-httpcache signaler works).
I'm not aware (at least from my perspective) of another use cache - I'd like to simply have more than one instance of varnish working and being able to propagate BAN / PURGE request to all of them.
Ah, I see. So even creating a plugin as opposed to a service that's always running wouldn't necessarily help with your use case (as you'd still need kubectl). I think an external solution such as http-broadcast would be preferable although it doesn't look like http-broadcast is built for kubernetes (not sure how you'd get endpoint updates in particular). I'm curious to see if there are any "broadcast" type solutions like this out there. If there's not, maybe we should create a separate repo and start it?
https://github.com/FUSAKLA/k8s-service-broadcasting looked promising but it hasn't been touched in 3 years.
Hi folks, this is a pretty common use case, we'll do a pilot with https://github.com/FUSAKLA/k8s-service-broadcasting (thanks for the pointer @cin).
I hope I can post here our findings in a week or so
Awesome @NITEMAN! Let us know how it goes. TBH, I'm surprised there doesn't appear to be any maintained projects out there for broadcasting HTTP requests to pods behind a service. I would actually like something that would allow parallelization of requests for shell commands, Redis, and Cassandra as well. Both of the DBs have CLIs that return results depending on that pod (i.e. every pod has a unique view of certain stats or w/e). I currently have a set of scripts that I use to broadcast these commands (in parallel) to pods that match a label. Obviously that doesn't work here bc it has to be run from my laptop (not in a cluster). I started going down the road of looking into a pluggable solution for this problem but it's definitely not something I have time for right now. Furthermore, dependency injection in Golang scares me a bit. ;) Mostly bc I've never had a chance to use it.
We'll see ;)... a project without recent commits is not necessarily unmaintained, it can just work
Before reaching this issue, my mental approach were leveraging https://doc.traefik.io/traefik/routing/services/#mirroring-service (since it's already in place as cluster's ingress controller) or mixing https://www.haproxy.com/blog/haproxy-traffic-mirroring-for-real-world-testing/ with some kind of service discovery (since I'm quite familiar with haproxy), but k8s-service-broadcasting seems to fit just better and since it's FOSS we can always contribute back if we extend/fix/patch it
Our first test with k8s-service-broadcasting looks promising, just a matter of customizing a bit the example and it works like charm. We have even set up an Ingrees Route allowing a remote system to purge the cluster
That's great news @NITEMAN! Thanks for spending the time to test this and confirm. @jsobiecki FYI.
Thanks! I'll check in next week :)
Sorry for late answer, been busy :) I'd like to write - that k8s-service-broadcasting with small customization works like a charm.
Glad to hear @jsobiecki! Going to close this for now.
May a mention on de docs with an example be worthy for others?
Sure, maybe add a new section describing what needs to be done?