hull: A tool to measure command performance

What hull aims to achieve is the performance measurement of shell commands. It does so by measuring the time elapsed between invocation and termination of a program invoked, and emits the measurements for collection.

hull is a command wrapper; it wraps around a shell invocation.

Performance is paramount and is one of the goals of this project. It is therefore suited both for interactive and non-login shell session usages.

In order to run fast, hull emits its data collection over UDP.

It sends a JSON encoded payload over UDP for a listener to persist or relay. A good companion for hull is transponder: https://github.com/oyiptong/transponder

By default hull emits to 127.0.0.1:48656. It runs on Mac OS and Linux.

Usage

hull is not invoked directly. It relies on path interception to function. Essentially, it figures out the name of the program to run by looking at its filename and removes itself from the path and executes the target program.

To function, hull requires two things:

command whitelist
path interception

Command whitelist

To use hull, create symlinks in its HULL_ROOT directory. By default, this is /etc/hull. You can define a custom root directory by setting the HULL_ROOT environment variable.

In effect, the HULL_ROOT serves both as a whitelist of commands to be observed and an invocation mechanism.

You simply need to create symlinks in the name of the program and place them in this directory.

Path interception

For hull to get invoked, the PATH environment variable needs to include the HULL_ROOT before any other PATH value. One would set it as the last thing in .bashrc, .zshrc, .profile, etc.

Performance

Because we expect stats to be sent every time a user executes a shell command, performance is key. The goal is to have the monitoring take strictly less than 10ms overall.

This has been benchmarked on an 8-core AWS machine (Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz) in two situations:

command wrapped with hull
command unwrapped (nowrap)

The shell command used looked like:

for i in {1..5000}; do (time ls) 2>> wrapped.log 1>/dev/null; done

The results are as follows:

status	mean	median
nowrap.log	0.0031828	0.003
wrapped.log	0.0063196	0.006

This means that over 5k runs of ls in the same directory, with the only difference between the 2 different kinds of runs was the use of hull, there was a ~3 ms difference in both mean or median.

Payload Schema

The data returned is serialized in JSON. While binary serialization would've been faster and more compact, the performance gains aren't big enough in the grand scheme of things.

Should this change, another payload format could be used in the future.

hull returns an EventsPayload, which is a JSON object containing only one property: events. In turn, events has as values an array of Events.

There are 2 possible Event types emitted:

hull_timing
hull_fatal_error

Events are structured as follows:

{
  event_name: "example",
  event_data: {
    ...
  }
}

The EventsPayload which is emitted by hull hence looks like:

{
  events: [
    {
      event_name: "example",
      event_data: {
        ...
    },
    ...
  ]
}

hull_timing

In the case of the hull_timing event, hull will emit an Event with the following data payload:

{
  events: [
    {
      event_name: "hull_timing",
      event_data: {
        cmd: <command name>,
        args: ["a", "list", "of", "parameters"],
        run: <time in millis>,
        created_at: <timestamp in seconds>,
        status_code: <program exit code>
      }
    }
  ]
}

hull_fatal_error

This Event is only to inform that there was something incorrect in hull's execution. It only contains the timestamp of the invocation.

Example Setup

In another shell, one needs to run a receiver for the datagrams listening on 127.0.0.1:48656.

Then one can do the following to setup hull:

$ pwd
/home/oyiptong/hull
$ sudo mkdir /etc/hull
$ sudo ln -s /home/oyiptong/hull/target/release/hull /etc/hull/ls
$ export PATH="/etc/hull:$PATH"

After this setup, where the user has decided to measure the execution time of ls, one just needs to invoke ls. In fact, we can know hull is running by coercing it to log:

$ RUST_LOG=info ls
Cargo.lock     Cargo.toml     LICENSE-APACHE LICENSE-MIT    README.md      build.rs       src            target
INFO:hull: cmd: ls run:3.759448 ms telemetry: 0.227966 ms total: 3.963549 ms

Should one have a server listening, they would receive this data packet (albeit here prettified):

{'events': [{'event_data': {'args': ['-G'],
                            'cmd': 'ls',
                            'created_at': 1485755468,
                            'run': 4.316207,
                            'status_code': 0},
             'event_name': 'hull_timing'}]}

Build instructions

Requirements are:

Mac OS or Linux (BSD's may be supported, but are untested)
Rust 1.36.0

rustup is recommended for compiler management.

To install dependencies and build with compiler optimizations, simply run:

$ cargo build --release

The statically compiled binary will be found at:

./target/release/hull