ATAM4Ex - Acceptance Tests As Monitors - for EliXir

Provides a framework for repeatedly running ExUnit tests and exposing the results in a standard format.

The original ATAM4J Java framework was developed at the FT, and is used for monitoring SLAs. ATAM4Ex provides the same functionality as an Elixir project, which makes the acceptance tests more robust, avoiding memory leaks and stateful crashes.

Synopsis

Tests are written as standard ExUnit tests, and thus may be run stand-alone with mix test, as well as under the ATAM4Ex supervisor.

Tests are loaded from the standard test directory by default, and must be match pattern *_test.exs.

An application which is host to the acceptance tests starts the ATAM4Ex supervisor tree either using ATAM4Ex.init/1 and ATAM4Ex.start_link/1, or by using the ATAM4Ex.Application module in its registered Application module:

defmodule MyAppATs do
    use ATAM4Ex.Application, otp_app: :myapp_at
end

This makes the MyAppATs module a startable application which can be registered in mix.exs so that it starts when the BEAM VM starts:

def application do
[
    mod: {MyAppATs, []},
    extra_applications: [:logger]
]
end

Note that ATAM4Ex's own mix.exs has an extra_applications dependency on ex_unit - this is important for releases, else the ExUnit modules will not be included, since they are normally only for testing in :test. You don't need to put anything in your app's mix.exs, the release mechanism (e.g. Distillery, or mix release) should take care of it.

Configuration

The otp_app option specified to the ATAM4Ex.Application module allows specifying the root of the configuration, in this case the ATAM4Ex configuration will be loaded (e.g. via config.exs) such that it is available via Application.get_env(:myapp_at, :atam4ex):

config :myapp_at, :atam4ex,
    initial_delay_ms: 10_000,
    period_ms: 30_000,
    timeout_ms: 120_000,
    ex_unit: [max_cases: 2, timeout: 5_000, exclude: [category: :local]]

For details of all the configuration options, see the documentation for the ATAM4Ex module.

You can also specify configuration by providing a atam4ex_opts/1 function in the application module; this receives any configuration extracted from config.exs as a keyword list, and should return a keyword list containing an adjusted configuration:

defmodule MyAppATs do
    use ATAM4Ex.Application, otp_app: :myapp_at

    def atam4ex_opts(opts) do
      Keyword.merge(opts, [period_ms: 60000])
    end
end

This function executes at run-time, so can be used to parse environment variables etc.

Starting the scheduled tests

You can then start your application using mix with:

$ mix run --no-halt

(--no-halt keeps the application running long enough to schedule the tests)

or in IEx with:

iex -S mix

Test results will be reported, as normal, to the console, so details of failures etc. appear in the application logs.

Test results can also be obtained programatically by calling ATAM4Ex.Collector.results/0 (or ATAM4Ex.Collector.results/1), which is what the Web Server does.

Default Web server

Unless the server: false configuration option is given, ATAM4Ex will start an HTTP server (on port 8080 by default), configured with the list of options on the server configuration property. e.g. to serve TLS you might put the following in config.exs:

config :myapp_at, :atam4ex,
  server: [port: 8443, scheme: :https, certfile: "/path/to/certfile", keyfile: "/path/to/keyfile"]

See Plug.Cowboy for details of all the options, including those for running under TLS.

The server responds to requests on the /tests path.

Response Format

The server serves test results in ATAM4J's JSON format, which is a list of tests, each with a passed flag, and an overall status, which is "ALL_OK" if all tests are passing, e.g.

$ curl 'http://localhost:8080/tests'
{"tests":[{"testName":"test user by id","testClass":"Elixir.GraphqlATTest","passed":true,"category":"critical"}],"status":"ALL_OK"}

Other values for status are TOO_EARLY, i.e. the tests haven't run yet, and FAILURES which mean some tests have failed.

The HTTP status is always 200, regardless of results.

Test Environment

ATAM4Ex supports loading a test environment (AKA configuration) via YAML files. This optional feature allows different configurations to be used for various run-time environments, e.g. host names, and api-keys are likely to be different in staging and production environments, and is more flexible than using config.exs (which is intended for build environments, not deployment environments), and may be more convenient than using releases.exs because of the ability to provide values for different run-time environments, and do type conversions.

The ATAM4Ex.Environment module can load YAML files, by default from an env directory (which should be packaged with your app release). Usually you would do this in your test_helper.exs file:

# load environment specified in "${APP_ENV}.yaml", or "development.yaml"
ATAM4Ex.Environment.load_environment!(System.get_env("APP_ENV") || :development)
ExUnit.start

Tests can then access the environment settings via ATAM4Ex.Environment.config/0 (or config/1) at any point in their execution, e.g. during setup/1 to provide a context, or during the test itself.

The YAML parser provides a mechanism for resolving system environment variables via :env keys, e.g. if your env/production.yaml file contained:

system_url: https://my-system-prod
system_api_key:
    :env: SYSTEM_API_KEY
server_port:
    :env: PORT
    :as: integer

ATAM4Ex.Environment.config(:system_api_key) will be resolved at load time to be the value of the SYSTEM_API_KEY environment variable, and PORT would be parsed as an integer and be available via ATAM4Ex.Environment.config(:server_port).

Note that for convenience, map keys in the YAML are converted to atoms.

Installation

For the bleeding edge, latest and most unstable version, add to your mix.exs file:

def deps do
  [
    {:atam4ex, github: "Financial-Times/atam4ex"}
  ]
end

Dependencies

Most of ATAM4Ex's dependencies are optional; if you want to use the web-server, or parse YAML, you'll need to add explicit dependencies to your app's mix.exs file.

ATAM4Ex needs plug_cowboy and jason for running its 'built-in' web-server, and yaml_elixir for ATAM4Ex.Environment. If you aren't using one or both of those components, then you can leave the corresponding deps out.

Take a look at this project's mix.exs for compatible versions of the optional components, but as of the time of writing:

# for ATAM4Ex.Environment
{:yaml_elixir, "~> 2.4"},

# for http server (ATAM4Ex.Web etc.)
{:jason, "~> 1.0"},
{:plug_cowboy, "~> 2.0"},

Building a Release with Distillery

The point of ATAM is that your tests run continuously somewhere; for this purpose you probably want to make a stand-alone release, rather than relying on mix run --no-halt.

Here's how:

You need to copy your tests and environment files to the release, else it won't be able to find them. Assuming you are using the default test and env directories, add set overlays entries to your rel/config.exs:

release :myapp_at do
  set version: current_version(:myapp_at)
  set overlays: [
    {:mkdir, "test"},
    {:copy, "test", "test"},
    {:mkdir, "env"},
    {:copy, "env", "env"}
  ]
end

This copies the test and env dirs to the releases directory, see Overlays and Configuration in the Distillery docs.

That's it. Otherwise it's a standard Elixir application release.

Building a release with Elixir 1.9

Elixir's mix release command packages your application in the _build/${MIX_ENV}/dev/rel/ directory.

While Distillary supports overlays, you can achieve the same effect using release steps, which are defined in your mix.exs file. You will need to write a function to copy the test and env directories to the release, e.g.

  def project do
    ...
    releases: [
      default: [
        steps: [
          :assemble,
          &copy_test_files/1,
        ]
      ]
    ]
  end

  defp copy_test_files(release) do
    ["test", "env"]
    |> Enum.each(fn src ->
      dest = Path.join([release.path, src])
      File.mkdir_p!(dest)
      File.cp_r!(src, dest)
    end)
    release
  end

If you are using Docker, you could alternatively copy the test files using your Dockerfile; note that unless you specify an absolute test_dir (or env_dir), tests will always be looked for relative to the current working directory, i.e. ./test/*_test.exs.

exit9/atam4ex