Generalized support for statistics export to monitoring systems.
Closed this issue · 6 comments
How would you feel about removing direct folsom integration from fuse?
My system uses exometer for reporting, and unfortunately also riak_core (which is broken wrt. other applications depending on folsom, Cf. basho/riak_core#636).
Given that the fuse application already exports the state of a particular fuse via fuse:ask/2, and also fuse_event, it seems like there's enough exported to allow users to integrate fuse with whatever metrics system at a higher level.
Wrt. exporting counts of foo.melt, foo.ok, foo.blown, perhaps a call like
fuse:stats(Name :: atom()) -> [proplists:property()].
would be more general.
Testing could be done by including calls to fuse:stats/1 in the generated command sequences of fuse_eqc.erl, with a postcondition verifying that a particular fuse's (modeled) history is reflected accurately in the stats export.
What do you think?
First thought: we should invert the dependency! Fuse should not depend on folsom, but another system could depend on both folsom and fuse, or folsom and exometer. This way, we don't mandate a specific version of tracker and this improves the flexibility of the system as a whole. We then provide a "plug in" module in the configuration for these kinds of things and we call that module whenever we want to do things with stats. By reconfiguring the name of the plug-in module, one can easy define fuse_folsom, fuse_trace, fuse_et, fuse_exometer and so on, and use the right module.
The change is definitely worthwhile to pursue. It would require a major version bump, but I think that should be done. The other part is to rewrite documentation appropriately.
Sounds good; I'll take a shot at this.
When you say "the configuration", do you mean via:
StatsModule = application:get_env(fuse, stats_plugin, fuse_folsom)
or similar? That's not a problem to set up in the state of the fuse_server process, but any concerns with calls like this in the async_dirty fast path of fuse_server:ask_/1?
Finally, is "master" the right place to be working as of now? I'm getting errors with tests there, both from folsom's "bear" dependency requiring application:start and (with that fixed), in the QC tests:
https://gist.github.com/emauton/d33385c06e16b6446161
Regarding performance worries: I want somebody to have a problem before I try to solve it. So application:get_env/3 is definitely the way to go for now.
The master should be stable. If it is not, it is an error. You should be able to branch off of master at any point knowing you have a stable foundation on which to build patches.
The QC errors are worrisome. I should probably look into why that happens :)
d195fae is not a serious commit btw - just stashing away some exploratory work from earlier for reference.
Ok, I took the time to look at this. The EQC errors you are seeing are probably expected. To compile the code for EQC-analysis, you need to run make app_eqc since we are setting an EQC_TESTING compile time knob.
The knob overrides timing in the model in order to make sure we control time in the system. This is necessary to test correctness of parallel timing constructions and to forward time correctly in the model. If you compile without it, things go bad very quickly, and that is essentially what you see. In general,
fuse_eqc:r(seq, {1, min}),
fuse_eqc:r(par, {1, min}),
should work out-of-the-box, but the PULSE enabled variant requires a more careful recompile to pull off, and its setup is less stable.
All in all, we should hence be able to use the test cases on the new code as well.