Exometer Core - Erlang instrumentation package, core services

Version: Feb 17 2024 15:18:00

Authors: Ulf Wiger (ulf.wiger@feuerlabs.com), Magnus Feuer (magnus.feuer@feuerlabs.com).

The Exometer Core package allows for easy and efficient instrumentation of Erlang code, allowing crucial data on system performance to be exported to a wide variety of monitoring systems.

Exometer Core comes with a set of pre-defined monitor components, and can be expanded with custom components to handle new types of Metrics, as well as integration with additional external systems such as databases, load balancers, etc.

This document gives a high level overview of the Exometer system. For details, please see the documentation for individual modules, starting with exometer.

Note the section on Dependency Management for how to deal with optional packages, both users and developers.

Table of Content

Concepts and Definitions

Exometer Core introduces a number of concepts and definitions used throughout the documentation and the code.

Metric

A metric is a specific measurement sampled inside an Erlang system and then reported to the Exometer Core system. An example metric would be "transactions_per_second", or "memory_usage".

Metrics are identified by a list of terms, such as given below:

[ xml_front_end, parser, file_size ]

A metric is created through a call by the code to be instrumented to exometer:new/2. Once created, the metric can be updated through exometer:update/2, or on its own initiative through the exometer_probe:sample behavior implementation.

Data Point

Each metric can consist of multiple data points, where each point has a specific value.

A typical example of data points would be a transactions_per_second (tps) metric, usually stored as a histogram covering the last couple of minutes of tps samples. Such a histogram would host multiple values, such as min, max, median, mean, 50_percentile, 75_percentile, etc.

It is up to the type of the metric, and the data probe backing that type (see below), to specify which data points are available under the given metric.

Metric Type

The type of a metric, specified when the metric is created through exometer:new/2, determines which exometer_entry callback to use.

The link between the type and the entry to use is configured through the exometer_admin module, and its associated exometer defaults configuration data.

The metric type, in other words, is only used to map a metric to a configurable exometer_entry callback.

Entry Callback

An exometer entry callback will receive values reported to a metric through the exometer:update/2 call and compile it into one or more data points. The entry callback can either be a counter (implemented natively in exometer), or a more complex statistical analysis such as a uniform distribution or a regular histogram.

The various outputs from these entries are reported as data points under the given metric.

Probe

Probes are a further specialization of exometer entries that run in their own Erlang processes and have their own state (like a gen_server). A probe is implemented through the exometer_probe behavior.

A probe can be used if independent monitoring is needed of, for example, /proc trees, network interfaces, and other subsystems that need periodic sampling. In these cases, the exometer_probe:probe_sample() call is invoked regularly by exometer, in the probe's own process, in order to extract data from the given subsystem and add it to the metric's data points.

Caching

Metric and data point values are read with the exometer:get_value/1 function. In the case of counters, this operation is very fast. With probes, the call results in a synchronous dialog with the probe process, and the cost of serving the request depends on the probe implementation and the nature of the metric being served.

If the cost of reading the value is so high that calling the function often would result in prohibitive load, it is possible to cache the value. This is done either explicitly from the probe itself (by calling exometer_cache:write/3), or by specifying the option {cache, Lifetime} for the entry. If an entry has a non-zero cache lifetime specified, the get_value/1 call will try fetching the cached value before calling the actual entry and automatically caching the result.

Note that if {cache, Lifetime} is not specified, exometer:get_value/1 will neither read nor write to the cache. It is possible for the probe to periodically cache a value regardless of how the cache lifetime is set, and the probe may also explicitly read from the cache if it isn't done automatically.

Subscriptions and Reporters

The subscription concept, managed by exometer_report allows metrics and their data points to be sampled at given intervals and delivered to one or more recipients, which can be either an arbitrary process or a Reporter plugin.

Each subscription ties a specific metric-datapoint pair to a reporter and an interval (given in milliseconds). The reporter system will, at the given interval, send the current value of the data point to the subscribing reporter. The subscription, with all its parameters, is setup through a call to exometer_report:subscribe/4.

In the case of processes, subscribed-to values will be delivered as a message. Modules, which implement the exometer_report callback behavior, will receive the plugins as a callbacks within the exometer_report process.

Subscriptions can either be setup at runtime, through exometer_report:subscribe/4 calls, or statically through the exometer_report configuration data.

Built-in entries and probes

There are a number of built-in entries and probes shipped with the Exometer Core package, as described below:

counter (exometer native)

The counter is implemented directly in exometer to provide simple counters. A call to exometer:update/2 will add the provided value to the counter.

The counter can be reset to zero through exometer:reset/1.

The available data points under a metric using the counter entry are value and ms_since_reset.

fast_counter (exometer native)

A fast counter implements the counter functionality, through the trace_info system, yielding a speed increase of about 3.5 in comparison to the regular counter.

The tradeoff is that running tracing and/or debugging may interfere with the counter functionality.

A call to exometer:update/2 will add the provided value to the counter.

The counter can be reset to zero through exometer:reset/1.

The available data points under a metric using the fast_counter entry are value and ms_since_reset.

gauge (exometer native)

The gauge is implemented directly in exometer to provide simple gauges. A call to exometer:update/2 will set the gauge's value to the provided value. That is, the value of the gauge entry is always the most recently provided value.

The gauge can be reset to zero through exometer:reset/1.

The available data points under a metric using the gauge entry are value and ms_since_reset.

exometer_histogram (probe)

The histogram probe stores a given number of updates, provided through exometer:update/2, in a histogram. The histogram maintains a log derived from all values received during a configurable time span and provides min, max, median, mean, and percentile analysis data points for the stored data.

In order to save memory, the histogram is divided into equal-sized time slots, where each slot spans a settable interval. All values received during a time slot will be averaged into a single value to be stored in the histogram once the time slot expires. The averaging function (which can be replaced by the caller), allows for high-frequency update metrics to have their resolution traded against resource consumption.

exometer_uniform (probe)

The uniform probe provides a uniform sample over a pool of values provided through exometer:update/2. When the pool reaches its configurable max size, existing values will be replaced at random to make space for new values. Much like exometer_histogram, the uniform probe provides min, max, median, mean, and percentile analysis data points for the stored data.

exometer_spiral (probe)

The spiral probe maintains the total sum of all values stored in its histogram. The histogram has a configurable time span, all values provided to the probe, through exometer:update/2, within that time span will be summed up and reported. If, for example, the histogram covers 60 seconds, the spiral probe will report the sum of all values reported during the last minute.

The grand total of all values received during the lifetime of the probe is also available.

exometer_function [entry]

The function entry allows for a simple caller-supplied function to be invoked in order to retrieve non-exometer data. The exometer_function:get_value/4 function will invoke a Module:Function(DataPoints) call, where Module and Function are provided by the caller.

The function entry provides an easy way of integrating an external system without having to write a complete entry.

Built in Reporters

Exometer Core ships with some built-in reporters which can be used to forward updated metrics and their data points to external systems. They can also serve as templates for custom-developed reporters.

exometer_report_tty

The exometer_report_tty reporter is mainly intended for experimentation. It outputs reports directly to the tty.

Instrumenting Erlang code

The code using Exometer Core needs to be instrumented in order to setup and use metrics reporting.

Exometer Core Start

The system using Exometer Core must start the exometer application prior to using it:

application:start(lager),
application:start(exometer_core).

Note that dependent applications need to be started first. On newer OTP versions (R61B or later), you can use application:ensure_all_started(exometer).

For testing, you can also use exometer:start/0.

See Configuring Exometer Core for details on configuration data format.

Creating metrics

A metric, can be created throuh a call to

exometer:new(Name, Type)

Name is a list of atoms, uniquely identifying the metric created. The type of the metric, specified by Type will be mapped to an exometer entry through the table maintained by exometer_admin Please see the Configuring type - entry maps for details.

The resolved entry to use will determine the data points available under the given metric.

Deleting metrics

A metric previously created with exometer:new/2 can be deleted by exometer:delete/1.

All subscriptions to the deleted metrics will be cancelled.

Setting metric values

A created metric can have its value updated through the exometer:update/2 function:

exometer:update(Name, Value)

The Name parameter is the same atom list provided to a previous exometer:new/2 call. The Value is an arbitrarty element that is forwarded to the exometer:update/2 function of the entry/probe that the metric is mapped to.

The receiving entry/probe will process the provided value and modify its data points accordingly.

Retrieving metric values

Exometer-using code can at any time retrieve the data point values associated with a previously created metric. In order to find out which data points are available for a metric, the following call can be used:

exometer:info(Name, datapoints)

The Name parameter is the same atom list provided to a previous exometer:new/2 call. The call will return a list of data point atoms that can then be provided to exometer:get_value/1 to retrieve their actual value:

exometer:get_value(Name, DataPoint)

The Name paramer identifies the metric, and DataPoints identifies the data points (returned from the previous info() call) to retrieve the value for.

If no DataPoints are provided, the values of a default list of data points, determined by the backing entry / probe, will be returned.

Setting up subscriptions

A subscription can either be statically configured, or dynamically setup from within the code using Exometer Core. For details on statically configured subscriptions, please see Configuring static subscriptions.

A dynamic subscription can be setup with the following call:

exometer_report:subscribe(Recipient, Metric, DataPoint, Inteval)

Recipient is the name of a reporter.

Set metric options

Each created metric can have options setup for it through the following call:

exometer:setopts(Name, Options)

The Name paramer identifies the metric to set the options for, and Options is a proplist ([{ Key, Value },...]) with the options to be set.

Exometer Core looks up the the backing entry that hosts the metric with the given Name, and will invoke the entry's setopts/4 function to set the actual options. Please see the setopts/4 function for the various entries for details.

Configuring Exometer Core

Exometer Core defaults can be changed either through OTP application environment variables or through the use of Basho's cuttlefish (https://github.com/basho/cuttlefish).

Note: Exometer Core will check both the exometer and the exometer_core application environments. The exometer environment overrides the exometer_core environment. However, if only exometer_core is used, any exometer environment will simply be ignored. This is because of the application controller: environment data is not loaded until the application in question is loaded.

Configuring type - entry maps

The dynamic method of configuring defaults for exometer entries is:

exometer_admin:set_default(NamePattern, Type, Default)

Where NamePattern is a list of terms describing what is essentially a name prefix with optional wildcards ('_'). A pattern that matches any legal name is ['_'].

Type is an atom defining a type of metric. The types already known to exometer, counter, fast_counter, ticker, uniform, histogram, spiral, netlink, and probe may be redefined, but other types can be described as well.

Default is either an #exometer_entry{} record (unlikely), or a list of {Key, Value} options, where the keys correspond to #exometer_entry record attribute names. The following attributes make sense to preset:

{module, atom()}              % the callback module
{status, enabled | disabled}  % operational status of the entry
{cache, non_neg_integer()}    % cache lifetime (ms)
{options, [{atom(), any()}]}  % entry-specific options

Below is an example, from exometer_core/priv/app.config:

{exometer, [
    {defaults, [
        {['_'], function , [{module, exometer_function}]},
        {['_'], counter  , [{module, exometer}]},
        {['_'], histogram, [{module, exometer_histogram}]},
        {['_'], spiral   , [{module, exometer_spiral}]}
    ]}
]}

In systems that use CuttleFish, the file exometer/priv/exometer.schema contains a schema for default settings. The setup corresponding to the above defaults would be as follows:

exometer.template.function.module  = exometer_function
exometer.template.counter.module   = exometer
exometer.template.histogram.module = exometer_histogram
exometer.template.spiral.module    = exometer_spiral

Configuring statically defined entries

Using the exometer environment variable predefined, entries can be added at application startup. The variable should have one of the following values:

{script, File} - File will be processed using file:script/2. The return value (the result of the last expression in the script) should be a list of{Name, Type, Options} tuples.
{apply, M, F, A} - The result of apply(M, F, A) should be {ok, L} whereL is a list of {Name, Type, Options} tuples.
L, where L is a list of {Name, Type, Options} tuples or extended instructions (see below).

The list of instructions may include:

{delete, Name} - deletes Name from the exometer registry.
{select_delete, Pattern} - applies a select pattern and deletes all matching entries.
{re_register, {Name, Type, Options}} - redefines an entry if present, otherwise creates it.

Exometer Core will also scan all loaded applications for the environment variables exometer_defaults and exometer_predefined, and process as above. If an application is loaded and started after exometer has started, it may call the function exometer:register_application() or exometer:register_application(App). This function will do nothing if exometer isn't already running, and otherwise process the exometer_defaults and exometer_predefined variables as above. The function can also be called during upgrade, as it will re-apply the settings each time.

Configuring static subscriptions

Static subscriptions, which are automatically setup at exometer startup without having to invoke exometer_report:subscribe/4, are configured through the report sub section under exometer.

Below is an example, from exometer/priv/app.config:

{exometer, [
    {report, [
        {subscribers, [
            {exometer_report_collectd, [db, cache, hits], mean, 2000, true},
            {exometer_report_collectd, [db, cache, hits], max, 5000, false}
        ]}
    ]}
]}

The report section configures static subscriptions and reporter plugins. See Configuring reporter plugins for details on how to configure individual plugins.

The subscribers sub-section contains all static subscriptions to be setup att exometer applications start. Each tuple in the prop list should be of one of the following formats:

{Reporter, Metric, DataPoint, Interval}
{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics}
{Reporter, Metric, DataPoint, Interval, RetryFailedMetrics, Extra}
{apply, {M, F, A}}
{select, {MatchPattern, DataPoint, Interval [, Retry [, Extra] ]}}

In the case of {apply, M, F, A}, the result of apply(M, F, A) must be a list of subscribers tuples.

In the case of {select, Expr}, a list of metrics is fetched using exometer:select(MatchPattern), where the result must be on the form {Key, Type, Status} (i.e. what corresponds to '$_'). The rest of the items will be applied to each of the matching entries.

The meaning of the above tuple elements is:

Reporter :: module()
Specifies the reporter plugin module, such asexometer_report_collectd that is to receive updated metric's data points.
Metric :: [atoms()]
Specifies the path to a metric previously created with anexometer:new/2 call.
DataPoint :: atom() | [atom()]'
Specifies the data point within the given metric to send to the receiver. The data point must match one of the data points returned byexometer:info(Name, datapoints) for the given metrics name.
Interval :: integer()' (milliseconds)
Specifies the interval, in milliseconds, between each update of the given metric's data point. At the given interval, the data point will be samples, and the result will be sent to the receiver.
RetryFailedMetrics :: boolean()
Specifies if the metric should be continued to be reported even if it is not found during a reporting cycle. This would be the case if a metric is not created by the time it is reported for the first time. If the metric will be created at a later time, this value should be set to true. Set this value to false if all attempts to report the metric should stop if when is not found. The default value is true.
Extra :: any()
Provides a means to pass along extra information for a given subscription. An example is the syntax option for the SNMP reporter, in which case Extra needs to be a property list.

Example configuration in sys.config, using the {select, Expr} pattern:

[
 {exometer, [
             {predefined,
              [{[a,1], counter, []},
               {[a,2], counter, []},
               {[b,1], counter, []},
               {[c,1], counter, []}]},
             {report,
              [
               {reporters,
                [{exometer_report_tty, []}]},
               {subscribers,
                [{select, {[{ {[a,'_'],'_','_'}, [], ['$_']}],
                           exometer_report_tty, value, 1000}}]}
              ]}
            ]}
].

This will activate a subscription on [a,1] and [a,2] in the exometer_report_tty reporter, firing once per second.

Configuring reporter plugins

The various reporter plugins to be loaded by exometer are configured in the report section under reporters

Each reporter has an entry named after its module, and the content of that entry is dependent on the reporter itself. The following chapters specifies the configuration parameters for the reporters shipped with exometer.

Creating custom exometer entries

Please see @see exometer_entry documentation for details.

Creating custom probes

Please see @see exometer_probe documentation for details.

Creating custom reporter plugins

Please see @see exometer_report documentation for details.

Customizing rebar.config

The OS environment variables EXOMETER_CORE_CONFIG_PREPROCESS and EXOMETER_CORE_CONFIG_POSTPROCESS can be used to insert a script, similar to rebar.config.script in the processing flow of the exometer build.

As the names imply, the script given by EXOMETER_CONFIG_CONFIG_PREPROCESS (if any) will be run before exometer does any processing of its own, and the EXOMETER_CORE_CONFIG_POSTPROCESS script (if any) will be run after all other processing is complete.