VMware VeloCloud SD-WAN Orchestrator API: Detect and Alert of WAN Anomaly

This Python app is containerised with Docker Compose for a modular and cloud native deployment that fits in any microservice architecture.

It does the following:

Call the VMware VeloCloud Orchestrator (VCO) API to retrieve the WAN quality metrics i.e. upload and download latency, jitter and packet loss, between the SD-WAN Edge and its associated SD-WAN Gateway, for all of the Edges in the enterprise network;
Detect WAN anomaly by comparing the WAN quality metrics of the last 5 minutes, to those of the 60 minutes before, with a sampling interval of 5 minutes; and
Send an email alert should the averages of the WAN quality metrics of the last 5 minutes be more than two standard deviations higher than the averages of the 60-minute historical baseline.

For details on the WAN path monitoring mechanism and sampling interval, please refer to the VMware SD-WAN Dynamic Multipath Optimisation (DMPO) article on the VMware SD-WAN Knowledge Base.

A detailed walk-through is available here.

Getting Started

Get started in three simple steps:

Download a copy of the app;
Create the environment variables for the VCO authentication and for email notification, and modify the sampling durations and interval and the crontab if needed; and
Docker Compose or build and run the image manually to start the app, or alternatively run the Python script as a standalone service.

Git Clone

Download a copy of the app with git clone. Be sure to pass the --recurse-submodules argument to initialise and update each submodule in the repository.

$ git clone --recurse-submodules https://github.com/kurtcms/vco-api-wan-anomaly-alert /app/vco-api-wan-anomaly-alert/

Environment Variables

The app expects the hostname, the API token or the username and password for the VCO; as well as the SMTPS port number, SMTP server address, the alert receiver email address, the alert sender email address and password; as environment variables in a .env file in the same directory.

Should both the API token, and the username and password, for the VCO be present, the app will always use the API token.

Be sure to create the .env file.

$ nano /app/vco-api-wan-anomaly-alert/.env

And define the variables accordingly.

VCO_HOSTNAME = 'vco.managed-sdwan.com/'

# Either the API token
VCO_TOKEN = '(redacted)'

# Or the username and password
VCO_USERNAME = 'kurtcms'
VCO_PASSWORD = '(redacted)'

# For email notification
EMAIL_SSL_PORT = 465
EMAIL_SMTP_SERVER = 'smtp.kurtcms.org'
EMAIL_SENDER = 'alert@kurtcms.org'
EMAIL_RECEIVER = 'noc@kurtcms.org'
EMAIL_SENDER_PASSWORD = '(redacted)'

Sampling Durations and Interval

The intervals for the WAN quality metrics are 300 seconds i.e. 5 minutes and 3,600 seconds i.e. 60 minutes, for the present and historical baseline respectively, with a sampling interval of 300 seconds i.e. 5 minutes. All of these are passed to the respective function as argument at runtime and may be adjusted if needed.

$ nano /app/vco-api-wan-anomaly-alert/vco_api_wan_anomaly_alert.py

Modify the values as appropriate.

conn.detect_wan_anomaly(min_per_sample = 5,
    interval_sec_present = 300,
    interval_sec_hist = 3600)
'''
min_per_sample of 5 i.e. one sample every 5 minutes
interval_sec_present of 300 i.e. 5 minutes
interval_sec_hist of 3600 i.e. 60 minutes
'''

Crontab

By default the app is scheduled with cron to retrieve the WAN quality metrics every 5 minutes, with stdout and stderr redirected to the main process for Docker logs.

Modify the crontab if a different schedule is required.

$ nano /app/vco-api-wan-anomaly-alert/crontab

Docker Container

Packaged as a container, the app is a standalone, executable package that may be run on Docker Engine. Be sure to have Docker installed.

Docker Compose

With Docker Compose, the app may be provisioned with a single command.

Install Docker and Docker Compose with the Bash script that comes with app.

$ chmod +x /app/vco-api-wan-anomaly-alert/docker-compose/docker-compose.sh \
    && /app/vco-api-wan-anomaly-alert/docker-compose/docker-compose.sh

Start the containers with Docker Compose.

$ docker-compose -f /app/vco-api-wan-anomaly-alert/docker-compose.yml up -d

Stopping the container is as simple as a single command.

$ docker-compose -f /app/vco-api-wan-anomaly-alert/docker-compose.yml down

Build and Run

Otherwise the Docker image can also be built manually.

$ docker build -t vco_api_wan_anomaly_alert /app/vco-api-wan-anomaly-alert/

Run the image with Docker once it is ready.

$ docker run -it --rm --name vco_api_wan_anomaly_alert vco_api_wan_anomaly_alert

Standalone Python Script

Alternatively the vco_api_wan_anomaly_alert.py script may be deployed as a standalone service.

Dependencies

In which case be sure to install the following required libraries for the vco_api_main.py:

Install them with pip3:

$ pip3 install requests python-dotenv numpy pandas

Cron

The script may then be executed with a task scheduler such as cron that runs it once every 5 minutes for example.

$ (crontab -l; echo "*/5 * * * * /usr/bin/python3 /app/vco-api-wan-anomaly-alert/vco_api_wan_anomaly_alert.py") | crontab -

Email Alert

Email alert will be sent from EMAIL_SENDER to EMAIL_RECEIVER should an anomaly be found. The subject of the email will be WAN Anomaly Alert with the details of the anomaly in the email body.

Latency (download, ms) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 100.0 and is 2 standard deviation(s) away from the mean of 75.0 and standard deviation of 10.0 of the 60.0 minute(s) before.
Jitter (download, ms) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 5.0 and is 2 standard deviation(s) away from the mean of 2.0 and standard deviation of 1.0 of the 60.0 minute(s) before.
Packet Loss (download, %) of WAN BT Business Broadband between Edge LDN-vVCE and its associated Gateway is found to be 5.0 and is 2 standard deviation(s) away from the mean of 1.0 and standard deviation of 1.0 of the 60.0 minute(s) before.

kurtcms/vco-api-wan-anomaly-alert