Flaw in using broker sourced log dir usage data.
robobario opened this issue · 2 comments
I know the logic here hasn't changed, but I realised there is possibly a flaw here.
admin.describeCluster().nodes()
returns the currently alive nodes in the cluster.- So the logic here can't account for brokers which are not currently alive.
- So a broker with a nearly full disk that restarts will be forgotten about:
- Maybe broker 0 got to 99% full and the quota kicks in preventing writes to any brokers in the cluster. What happens if broker 0 is now restarted?
- It will disappear from the
admin.describeCluster()
result (for a time at least), which will release the throttling, allowing appends to the other brokers. Broker 0 might be down for a while if it needed to do log recover on restart. - When broker 0 does rejoin, there is a window before the quotas will get applied again. In the mean time there are appends it needs to replicate, so it fetches and appends and fills its disks.
I think this is all pretty unlikely. In practice it requires a number of things to align, but I think this is possible.
Originally posted by @tombentley in #37 (comment)
We could enable the plugin to operate on stale data for some configurable period. So we retain some cached data about the last state for each node and when calculating the next throttle factor we could combine the data scraped from Kafka with the last cached data. So if we scrape data for nodes [0, 2] and have cached data for [0, 1 ,2, 3] from 1 minute ago, we could proceed with the fresh description of [0,2] and the cached description of [1, 3].
After the configurable duration we could stop considering the cached data.
Could operating on stale data also be a problem? Maybe in some odd case like removing a full broker and replacing it with a new broker with more capacity we might continue blocking writes unneccesarily.