lobotmcj/johann

Lightweight and flexible scenario orchestration

Johann

Johann is a lightweight and flexible “scenario orchestrator”. It makes it easy to coordinate the actions of groups of computers into cohesive, reconfigurable scenarios. It’s sort of like the conductor for an orchestra of computers, and you get to write the music.

Contents

• Getting Started
• Usage
• This is an Alpha Prototype
• Roadmap
• Contributing
• Built With
• License
• Acknowledgments
• More Info

Getting Started

First use git to clone Johann and change directories into the newly cloned repo.

Installing

Johann is designed to be used on Linux and run in docker containers. It has been tested on Ubuntu 18.04, and likely works on several other distributions as well.

Johann requires the following to run:

• Docker Engine
• Docker Compose
• Make

Here is an example of how to install these on Ubuntu/Debian:

# Install make
sudo apt-get update
sudo apt-get install build-essential

# Install docker via convenience script (not for production environments)
curl https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
# log out and log back in
docker run hello-world

# Install docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

Running

Johann uses make to handle building and deploying its docker images. This can take a while the first time.

make dev

Usage

Johann orchestrates scenarios through reconfigurable YAML files called "Scores". Similar to musical scores, these files describe the the actions that the "Players" must take to perform their part/role in the scenario. Each Player's part in the scenario consists of "Measures" -- specific tasks and timing that Players must perform. Again similar to musical scores, the score file weaves these Measures together to orchestrate the Players in the full scenario. Each Player is currently comprised of a group of Docker container(s). In the future, these could be VMs or physical machines as well.

Lexicon

• Score -> Scenario/script describing the actions that Players must take and when
• Player -> Group of Docker container(s) that play the same part/role in the scenario
• Host -> An actual compute resource such as a Docker container, VM, or physical machine.
• Measure -> Maps a Task and its timing/configuration to Player(s) in a Score
• Task -> Specific action taken by Player(s) as part of a Measure

Example Score

The following is an example Score consisting of one Player ("docker_targets") with two Hosts which are both Docker containers (blank_3.6_buster and blank_3.7_buster). Note that the specific Hosts need not be specified in the Score file, and can be provided (or changed) at runtime either via API or GUI. Both of these specific Hosts are stock Debian containers with Python installed -- versions 3.6 and 3.7, respectively Neither of these containers have Johann installed -- it will installed or updated ("pushed") at runtime to match the version of Johann installed on the machine running Docker. This example Score has just one Measure executed by one Player (2 Hosts), with a Task to run the ls -la command in the root directory of each Host/container.

# Copyright (c) 2019-present, The Johann Authors. All Rights Reserved.
# Use of this source code is governed by a BSD-3-clause license that can
# be found in the LICENSE file. See the AUTHORS file for names of contributors.

---
name: test_johann_pushing
category: testing
description:
  test Johann's push functionality to install the player software on a host that doesn't
  have it already
players:
  docker_targets:
    name: docker_targets
    hosts:
      - blank_3.6_buster
      - blank_3.7_buster
    image: None
    scale: 2
measures:
  - name: ls_root
    players: [docker_targets]
    start_delay: 0
    task: johann.tasks_main.run_shell_command
    args:
      - "ls -la /"

• The name ls_root is the arbitrary name of the Measure and is used as a key in the API to interact with the Measure.
• The players key specifies which Players from the list defined above should perform this particular Measure.
• The Task johann.tasks_main.run_shell_command specifies a specific compatible action from the Johann tasks. See Johann's task code for a partial list of compatible Tasks.
  • Tasks are Python functions with the decorator @celery_app.task.
• The argument ls -la is supplied to the Task in this case to specify the command to be run as a shell command.

Running a Score

With the Johann Docker containers running via make dev, users can interact with Scores via either the command line or the web UI.

GUI

• Open a web browser and navigate to http://127.0.0.1/

• Click on the Scenarios tab to view the available Scores

• In the row containing the Score that you want to run, select one of the following options:

  • View: Displays the YAML and JSON representations of the Score file
  • Status: Displays the status of the current or last run depending on if a run in is progress
  • Launch: Runs the Score file
  • Reset: Resets the run to allow for a new run to be launched and monitored with status

• To run a Score select launch to be presented with the Launch Scenario screen.

• From this menu you can map available Hosts to the Players defined in the Score file.

  • Hosts can be added either via API or a file that is run at startup.

• Press the Launch Scenario button to launch the Score using the selected Hosts.

• This will automatically take you to the status page for the Score you just launched where you can watch the Measures of the Score play. Note: Some Scores, including the test score, may take a few minutes to initialize before running. This is where Johann is installing or updating itself on the Hosts.

• The status page also contains the raw output of the Tasks, in this case the ls -la command run on each container shown below.

Command Line

• To view available API endpoints

curl http://127.0.0.1:5000/

• To view available scores

curl http://127.0.0.1:5000/scores/

• To run a specific score

curl http://127.0.0.1:5000/affrettando/<score_name>

• To view the current status of a running score

watch 'curl http://127.0.0.1:5000/scores/<score_name>/status_short'

This is an Alpha Prototype

Johann is an evolutionary prototype in its initial development. It is not yet feature complete, and breaking changes can happen at any time. This is represented by its major version zero (0.y.z).

For now, Johann should be considered to be in perpetual alpha. This is made explicit by the "-alpha" or "a" in the version identifier. Please expect it to be rough around the edges (and maybe everywhere else).

Johann should only be used in isolated or protected networks of known and trusted hosts/users. It should only be used for research and development, and not in production.

Roadmap

Here are some planned improvements to Johann, in no particular order:

• add more documentation
• switch to pydantic
• switch to fastapi
• use mypy and pylint
• add more tests
• add user authentication
• support kwargs in Measures
• Score-level variables; configurable at runtime

Contributing

We welcome pull requests! Before starting, please communicate with us to discuss what you would like to change. Please also update tests and documentation as appropriate. Thanks!

Getting Started

Install development packages.

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install python3-dev python3-venv build-essential git

# Alpine (not officially supported)
apk add --no-cache python3-dev bash make git gcc linux-headers musl-dev

Setup the virtual environment used for Johann development. This also installs pre-commit hooks.

make dev-setup

Development Usage

Start Johann in development mode (this will also build it).

make dev

Testing

make test

Linting

Johann uses pre-commit. For the list of hooks used, see .pre-commit-config.yaml.

Lint the files staged for commit.

make lint

Lint all files in the repo.

make lint-all

Use safety to check for known dependency vulnerabilities.

make safety

Contribution Flow

Johann uses a variant of GitHub Flow based on squash/rebase, which is most closely described here. If you don't know how to squash or rebase yet, don't worry! We'll help you through it at PR time!

• Pull Requests should target the develop branch
• commits must have the Developer Certificate of Origin (DCO) (git commit -s)
  • DCO Wikipedia Article
• commits must also be PGP signed (git commit -S)
  • Instructions from GitHub
  • Instructions from Git

Again, if any of this is intimidating, don't worry about it! It was new to us too -- feel free to contact us or we can sort it out at PR time!

Pre-Commit Hooks

While they are opt-in, we strongly recommend the use of the pre-commit hooks provided (see #Linting). These hooks will need to pass before a PR is accepted, and it helps to work through them ahead of time. If you don't know how to or don't want to bother with a particular hook failure(s), that's perfectly fine, and we will help you out at PR time -- just prepend your git commit and/or push command with SKIP=[comma-separated list of hook names] or, as a last resort, append --no-verify.

Built With

In addition to the dependencies listed in Requirements, please see:

• setup.py
• requirements-dev.in
• .pre-commit-config.yaml

In addition to these, Johann is also made possible with the help of (alphabetically):

• gitlab-ci
• pmtr
• uwsgi-nginx-flask-docker

License

Use of this source code is governed by a BSD-3-clause license that can be found in the LICENSE file. See the AUTHORS file for names of contributors.

Acknowledgments

• JHU/APL for supporting Johann's licensing as open source
• Johann S.D.G.

More Info

More information is available at the docs, or you can contact someone from Johann S.D.G. .