Support dynamic configuration
jolestar opened this issue · 42 comments
Please support reload configuration files and do not need restart server.
I think there are two approach:
- Catch SIGHUP signal
- Offer a http api
Hi fluentbit, any idea or thinking about this request?
It would be great, if we can have this feature implemented.
SIGHUP currently takes down the process:
Fluent-Bit v0.13.4
Copyright (C) Treasure Data
[engine] caught signal (SIGHUP)
It would be great if this reloaded the config without taking down the process.
It would be nice to watch the configuration files as well and support hot configuration reloading as well.
If there is an API like prometheus /-/reload , it’s even better.
Or just use inotify to watch the confiug file and automatically reload. It's using inotify for the log anyway, right?
Not that I would do this, but I could imagine users modifying their config file on the server and writing out progress. If you reload on writes like that it may load an invalid or partial config. Also, this breaks from most server software that receive an explicit signal like SIGHUP.
What's the current problem of restarting Fluent Bit using systemd ?, I would like to learn the real need of config reload without restarting, as far as I can tell services that offer this always start a new process (behind the scenes).
The real complexity will be to add this handling on every single input plugin. Please elaborate the use-case.
@edsiper I'm running fluent-bit in a container on kubernetes, so it's not using systemd. Running another process manager in the container, at least on kubernetes, is an antipattern since you loose the process management features provided by kubernetes (restart/policies, monitoring, event log etc).
That being said, there is a case to be made for it being kubernetes's job to restart the fluent-bit container when the config changes. Something that's being discussed in kubernetes/kubernetes#22368
Currently the best option would be reloading on:
- inotify (people would need to write the config update atomically, but this is an established pattern and can be easily achieved by
mvthe file to the final location after creating it somewhere else) - @vinkdong's suggestion of an http endpoint which can be used with https://github.com/jimmidyson/configmap-reload as a sidecar container (it would watch the config file via inotify and send the http request to reload)
Reloading on SIGHUP should also work but requires an alpha feature: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/
@discordianfish The @edsiper's answer speaks volumes! I had so many problems with Fluent Bit that I switched to Filebeat, which is mature, written in Go so more people can contribute, and it has most of the features Fluent Bit is not even planning to have any time soon, including configuration reloading.
I think adding an inotify on the config and restart would be good.
the next step would be to emulate grafana and have multiple config maps, one for each pod that wants to have custom log message formats. This means that fluent-bit watches for new configmaps to be written, and reads them as they happen. (see https://github.com/coreos/prometheus-operator).
e.g. in my pod i could have a 'fluent-parser' and have a single parser entry, which would be dynamically loaded.
the restart can be ok as long as the trackingoffsets is used. Otherwise its disasterous.
@donbowman Filebeat already has multiple formats (let's say, Nginx access and error logs), and reloading works, too. Also, it has a super easy way to create your own parsing pipelines without any programming. And, most importantly, it has a very active and passionate team that works on it full-time. I'm sorry to be promoting Filebeat, but I invested almost one year in Fluent Bit and the return was only pain! It's not common knowledge, but Filebeat has very good built-in Kubernetes hints as well. The only built-in feature that is missing right now is a Prometheus metrics endpoint, but there's an open-source project, which I compiled into a 2MB-image sidecar, which exports Filebeat's rich JSON metrics to the Prometheus metrics format. And soon Filebeat will have native Prometheus metrics endpoint. Also, all the standard parsers feed namespaced metrics to Elasticsearch, so, you don't get messy metrics under the same name.
Hopefully, the Fluent Bit team reads this, looks into Filebeat, and starts putting more passion into delivering something that works with basic uses cases if not with more complex stuff.
If I was Treasure Data, I'd rewrite this in Go if not in Rust and make it work perfectly with Kubernetes and ignore all other stuff including my own niche platform. And then offer enterprise support plants. For example, why put configuration in files at the end of 2018, when it can read it directly from ConfigMaps, and watch them for updates?!
There's still a chance for this project as Elastic is door a tremendously poor job at advertising Filebeat and I don't think they still pay enough attention to Kubernetes.
I am not against configuration reloading, but understanding the why and it urgency helps to assign a priority, as you can guess with different interfaces in input, filters and outputs is not a straightforward change. As said above, for Kubernetes use-case config-reloading is an anti-pattern (immutable).
Now about Fluent Bit v/s other projects, I consider that is really good to learn from others, that's the way to go. We still have a plenty of room for improvements.
About programming languages, consensus is something that will unlikely exists and having different preferences is just fine. Users decide their tooling, hopefully they have choices, Fluent Bit is one of them.
@donbowman I don't think C, C++, Rust, and Go have much in common outside of being compiled languages.
@donbowman There are huge legacy codebase on C and C++ and this drives its "popularity". The Kubernetes ecosystem, which I personally care about, is dominated by Go. Here's a nice analysis of the subject by a very popular project, which analyzed the options in 2018 and on: https://blog.phusion.nl/2018/09/18/migrating-passenger-from-cxx-to-go/
The language discussion is entirely off-topic here. Could you move this somewhere else please?
Coming back to the original question, I'm wondering how I could use fluent-bit to support the following use case:
Customer is running a k8s cluster and wants to ship logs to an external "service". Service provisioning and binding follows the OSB API pattern, i.e. customer creates a service instance and then binds an "app" to that instance.
The result of the binding operation is a secret that fluent-bit would then (have to) digest to connect to the remote service instance.
One could, of course (like always) put an additional layer between fluent-bit and the remote service endpoint, e.g. have fluent-bit send to a local service proxy and make that service proxy listen to/watch the secret, but that doesn't sound like a good solution.
Any other opinions/suggestions?
I have an outstanding PR to fluent-bit #842 that should solve this [original] issue.
is the secret ok to have in your configmap for fluent-bit? In which case my PR achieves your goal.
Or is it something else that fluent-bit doesn't yet support?
where does the secret get injected, what is the output driver in use?
is the secret ok to have in your configmap for fluent-bit?
Yes
where does the secret get injected, what is the output driver in use?
Not sure whether I fully get this.
AFACT, the workflow that I described is what Service Catalog supposed to do.
The Docker output driver is json-file. Or did you mean something else?
i mean the fluent-bit output driver. e.g. how does the secret get injected.
Currently, it's out_http.
so presumably you end up with an auth-token that goes in a header, and want to use the Header feature of out_http with this JWT?
so some agent registers, gets the JWT, updates the fluent-bit config map, and fluent-bit auto-reloads that?
Another option would be to make fluent-bit config allow environment variable expansion, and get the token from env.
But, kubernetes/kubernetes#22368 this is also not solved in k8s yet as far as i know.
JWT auth token in Header feature of out_http? Where is that documented? Can't find it in https://fluentbit.io/documentation/current/output/http.html.
All I see there is HTTP_User and HTTP_Password.
Anyways, yes, it seems like there needs to be some sort of agent running, read auth data from secret, update config map, etc.
Or, sigh, have fluent-bit talk to local proxy and let the local proxy do that magic.
https://docs.fluentbit.io/manual/output/http
I was thinking the Header field, you can add the auth-header/jwt tag as needed there.
but you'd have to get it into the configmap.
I've changed the title of the issue since due to the feedback received internally and on GitHub what we need is dynamic configuration mechanisms which are more than a "config file reload".
After 1.0 we will focus on the design of this feature which involves having a parent process to monitor the engine plus other extensible interfaces to make this happens.
Hi @edsiper,
thanks for the update. Just out of curiosity: What does "after 1.0" mean in terms of dates?
hi @edsiper
is there any update?
Use case here: In Kubernetes, we configure fluentbit using a configmap. We have a multitenant platform where tenants can configure their own log outputs, whenever they change their log output configuration we update the config map, and then we need fluentbit to pick up this configuration somehow. Currently, the only way to do this is to update a dummy environment variable in the fluentbit DaemonSet, to trigger a full redeploy. It would be nice if fluentbit could pick up the change without a full redeploy. It doesn't even have to dynamically reconfigure itself, just watch the files, and if they change, exit. Kubernetes will restart the process.
Since k8s 1.15 you can use kubectl rollout restart to recreate pods so they pick up new configuration.
Much better than editing a dummy env var. 😄
we moved to use kustomize, it hashes the config-map name based on contents, thus achieving the rolling restart when changed.
Use case: We use Fluent Bit as a sidecar container in Kubernetes. Currently, making a minor change to our logging config requires us to restart all applications that use the sidecar.
Does Fluent Bit re-read Kubernetes Secrets at runtime?
Scenario: sending logs with tcp output and our ssl cert expires. We need to update the secret with the new cert. Will Fluent Bit pickup of the new secret during runtime without the need for a pod redeploy? Or is a redeploy required?
re-deploy is required. How often your SSL cert expires ?
re-deploy is required. How often your SSL cert expires ?
Approx every 6 months.
a rollout restart deployment is sufficient.
most people have 90 expiry, refreshed in 45-50 days, due to use of let's encrypt.
for the k8s sidecar use-case, especially when used with stateful applications that have a high cost for rolling the deployment, this is honestly not really an acceptable solution. IMO fluent-bit should be capable of detecting changes to the config file at run-time in some way, or at least provide an endpoint to update the config
Exactly, its nearly impossible to manage rollout restart command in bigger scales.., of course, You can always use https://github.com/stakater/Reloader, https://github.com/pusher/wave, or similar, but that should not be necessary..
FluentD is offering API for reloading, many projects are using it:
- https://github.com/vmware/kube-fluentd-operator/blob/master/config-reloader/fluentd/reloader.go#L31
- https://github.com/banzaicloud/logging-operator/blob/master/pkg/resources/fluentd/statefulset.go#L166
One thing is config change.. , its predictable, but secret (TLS) change is not ( Lets Encrypt renewal ).
Filebeat is offering live reload: https://www.elastic.co/guide/en/beats/filebeat/current/_live_reloading.html
Fluentbit-operator might address this issue. It facilitates the deployment of Fluent Bit and provides great flexibility in building a logging layer based on Fluent Bit. For now, it provides the following features:
- Fluent Bit Management: Deploy and destroy Fluent Bit Daemonset automatically.
- Custom Configuration: Select input/filter/output plugins via labels.
- Dynamic Reloading: Update configuration without rebooting Fluent Bit pods.
@edsiper I want to share another use case for configuration changes scan.
Imagine multiple teams working on different services. Each service should be deployed independently from potentially different team. Let's say that a service has some log filtering (grep like) requirements which the owner (some team) defines. Upon service deploy, the filter should be updated and ideally this should not cause rollout restart. It may happen multiple times per day to deploy a service. (of course changing the grep logic is less frequent)
That's a good candidate for configuration changes scan, unless you propose a better solution.
is there any update?
@edsiper @patrick-stephens is there any progress on this? If monitoring the config file is too expensive I'd be interested if a reload endpoint would be possible?
it was implemented with release 2.1.0, isn't it?
https://docs.fluentbit.io/manual/administration/hot-reload
would be great to add this in fluent-operator project