shovel_ansible

Install shovel + OpenObserve as systemd services, on a local dev VM or production VM.

Local Dev VM install

Start a local Lima VM

Mostly from Lima: a nice way to run Linux VMs on Mac

Install Lima:

brew install lima

Useful Lima commands:

limactl start default
lima  # open shell
limactl factory-reset default  # to test playbook from scratch
limactl stop

Install locally trusted TLS certificates

Uses mkcert - Note the warning!

Warning: the rootCA-key.pem file that mkcert automatically generates gives complete power to intercept secure requests from your machine. Do not share it.

brew install mkcert nss
mkcert -install
mkcert localhost

This generates ./localhost-key.pem and localhost.pem, signed by the rootCA cert generated by mkcert.

Install shovel + OpenObserve on the Lima VM

Note: if using an arm64 Mac, Lima VMs are aarch64 (another name for the same architecture), so get linux arm64/aarch64 versions of the releases.

Obtain a shovel release from GitHub or by building with GoReleasor

Get an OpenObserve release.

Update ./lima_vars.yaml with paths to the OpenObserve and shovel tar.gz files.

Export environmental variables for Ansible to read

export SHOVEL_SERVE_OPENOBSERVE_PASS='...';
export SHOVEL_SERVE_OPENOBSERVE_USER='...';
export ZO_ROOT_USER_EMAIL='...';
export ZO_ROOT_USER_PASSWORD='...';

I use envelope to manage environment variables, so I use the following commands to set up the environment:

envelope env create
envelope env var create --name ZO_ROOT_USER_EMAIL --value root@example.com
envelope env var create --name ZO_ROOT_USER_PASSWORD --value ZO_ROOT_USER_PASSWORD

# Re-use the root user password
envelope env ref create --name SHOVEL_SERVE_OPENOBSERVE_USER --ref-env-name "$PWD" --ref-var-name ZO_ROOT_USER_EMAIL
envelope env ref create --name SHOVEL_SERVE_OPENOBSERVE_PASS --ref-env-name "$PWD" --ref-var-name ZO_ROOT_USER_PASSWORD

export-env "$PWD"

Run the Ansible playbooks:

ansible-playbook \
    -i 'lima-default,' \
    --ssh-extra-args "-F $HOME/.lima/default/ssh.config" \
    --extra-vars "@lima_vars.yaml" openobserve.ansible.yaml shovel.ansible.yaml

Open sites

Open OpenObserve at: http://localhost:5080/web/traces?period=15m&query=&org_identifier=default

Open shovel at: https://localhost:8080/?count=1&nameservers=dns3.p09.nsone.net%3A53&protocol=udp&qnames=linkedin.com+www.linkedin.com&rtypes=A&subnetMap=&subnets=

Import dashboards

There isn't an API for this yet, so export by using the download icon when viewing a dashboard and import from the main menu.

Production VM

Obtain releases and export envvars as above.

Create prod_vars.yaml. Use ./lima_vars.yaml for reference

NOTE: ensure the -i flag for a single host ends with a comma (,)

ansible-playbook \
    --ask-become-pass \
    -i '<host>,' \
    --extra-vars "@prod_vars.yaml" \
    openobserve.ansible.yaml shovel.ansible.yaml

Debugging

# Check if the service is running
sudo systemctl status shovel

# Check logs
sudo journalctl -u shovel

# Analyze security to tighten systemd capabilities
sudo systemd-analyze security /etc/systemd/system/openobserve.service

bbkane/shovel_ansible