SASL started by default
Closed this issue · 8 comments
I'm coming from the Elixir world and when adding Fuse to my project SASL logs began showing up. This is the only Erlang dependency (of which there are many) for which this happens in my project. I'm curious as to why this is enabled by default and whether SASL logging could be specified in the application configuration.
Related discussion: https://elixirforum.com/t/addition-of-erlang-dep-causing-sasl-logging/9265/7
SASL is a necessary dependency in fuse.
The reason is fuse includes a fuse_monitor module which keeps track of broken fuses in your system. When a fuse is tripped, the alarm_handler is told about this. When the fuse repairs, the alarm is cleared. The alarm handler is part of the SASL package.
It is useful because you can then subscribe to events in the alarm handler and have your system react to broken fuses in a natural way. It allows you to have a loosely decoupled error handling between processes, since you don't have to "define" stuff inside fuse for this: just subscribe to the alarm handler and you are up and running.
To fix your immediate problem, you can configure the SASL application to avoid logging on the console. If memory serves, the knob is sasl_error_logger you need to set to false, but do check it yourself with erl -man sasl where it is documented. Alternatively, the usual Erlang solution is that the lager application can take over logging from SASL, but this is slightly more involved.
Terrific. Thanks for the color. I will take your first approach and set sasl_error_logger to false in my config: config :sasl, sasl_error_logger: :false
Nice it solves your problem!
To add: one problem with circuit breakers are that they make your system survive. Hence, attach a worker to the alarm_handler which forwards any broken fuse to the operations/devops console/office of your production system. This way, you know what internal fuses have triggered and this is often enough to know what subsystem is in trouble. I've had really good operational experience with this, as you often know a subsystem is in trouble right away, and you don't have to go analyzing what system is currently in a bad state: you get told.
This is something I'd be interested in using. My current experience with Fuse is via these two blog articles which just discuss the basic functionality. I'll need to look into how I'd use alarm_handler with Elixir. Right now I'm just logging when a Fuse is blown manually.
https://blog.mojotech.com/safeguard-web-service-failures-in-elixir-with-fuse/
http://www.rokkincat.com/blog/2015/09/24/circuit-breakers-in-elixir
Is the alarm handler fuse_event:add_handler/2? I'm looking at https://hexdocs.pm/fuse/. I see alarm_handler is in Erlang but am not sure how to utilize this feature 😬 . It's just what I'm looking for though.
See http://erlang.org/doc/man/alarm_handler.html
The alarm handler process is a
gen_eventevent manager process that receives alarms in the system. This process is not intended to be a complete alarm handler. It defines a place to which alarms can be sent. One simple event handler is installed in the alarm handler at startup, but users are encouraged to write and install their own handlers.
[...]
The default simple handler is calledalarm_handlerand it can be exchanged by callinggen_event:swap_handler/3 as gen_event:swap_handler(alarm_handler, {alarm_handler, swap}, {NewHandler, Args}).NewHandler:init({Args, {alarm_handler, Alarms}})is called. For more details, seegen_event(3)in STDLIB.
Just to make it clear: the fuse event handler is different from the alarm handler. Your systems might want to hook into events and react accordingly, so they are to use the fuse_event system. Whereas your deployed VM might want to watch and handle alarms of a higher kind, which is the purpose of the alarm_handler.
Both have their use cases. You also have stats collections inside fuse so you have a chance at seeing what is going on when things starts going bad. This is meant for exporting into e.g., Grafana.
So, for example, if I want to be notified when a fuse is blown I should create a custom event handler and add it using fuse_event:add_handler/2. Am I correct?