/openvidu-loadtest

OpenVidu load testing environment

Primary LanguageJavaApache License 2.0Apache-2.0

OpenVidu Load Test

WARNING: The Openvidu Load Test project is having some big changes in functionality and architecture, so the latest versions in the repository are not stable.

What you can do is clone the repository on the commit of one of the previous stable releases (or use the latest tags in the Releases):

  • KMS if you want to use emulation of browsers based on KMS.
  • Node if you want to use emulation of browsers based on Node webrtc (both versions work for real browsers).

After that, you will have to download the latest openvidu-browser (or the version of openvidu-browser you are using in your OpenVidu deployment) and add it to the directory at browser-emulator/public, and modify line 6 of browser-emulator/public/index.html to use the latest openvidu-browser.

This repository contains a distributed tool that allows you to perform a load test against an OpenVidu CE or an OpenVidu PRO deployment.

Take into account that you must to deploy OpenVidu platform before using this tool.


Table of Contents

  1. Project Architecture
  2. Usage instructions
  3. Sample test execution
  4. Analyze test results
  5. Browser Emulator documentation

Project Architecture

Load test architecture

  • Browser-emulator: Worker service implemented in NodeJS and controlled with a REST protocol that capable of connecting to an OpenVidu session and sending and receiving WebRTC media using openvidu-browser library. The browser-emulator service provides several connection modes:

    • Traditional mode: It is able to start and launch containerized Chrome browsers using Docker and Selenium emulating a fully real user.

    • Emulated mode: It is capable to emulate a user connection without use a browser:

      • Overriding a WebRTC API using node-webrtc library and getting the media from a canvas publishing a moving image.
      • Overriding a WebRTC API using node-webrtc library and getting the media from a video file using ffmpeg.
      • Overriding the peerConnection objects using Kurento and getting the media from a video file.

  • Loadtest Controller: Controller service in charge of the coordination of the browser-emulator workers. It read the load test scenario from a file and control the browser-emulator workers to connect participants loading OpenVidu platform.

Usage Instructions

These instructions assume that OpenVidu CE or PRO is already deployed. See how deploy OpenVidu PRO.

1. Launch workers

For testing locally

This instructions assume you are using a linux system. Windows and Mac can require some adaptation.


To start with the load test you will have to start a worker first and the execute the controller.

In the machines where you want to execute the workers you need to have NodeJS and Docker platform installed.

Then follow these steps:

  • Clone this repository
git clone https://github.com/OpenVidu/openvidu-loadtest.git
  • Start browser-emulator service (worker)
cd openvidu-loadtest/browser-emulator/
# Assuming that you're in a clean environment, the script will install:
# - NodeJS
# - Docker
# - Ffmpeg
# - Download te required media files for executing the containerized Chrome Browsers
sudo su
./prepare.sh
npm install
npm run start:dev-canvas
Running options

By default, this worker is listening on port 5000 that you have to specify later in the controller configuration. If you want to run it on another port just add SERVER_PORT=port_number before npm run start command:

SERVER_PORT=6000 npm run start

Moreover, the last run command will assign NODE_WEBRTC for emulated user types. You can run the following command for use KMS WebRTC:

npm run start:dev-kms
For testing on AWS
If you want to launch the load tests in a production environment, you can do that with AWS.

The browser-emulator services will be launched (by the loadtest-controller) in EC2 instances with the aim of emulate real users connected to OpenVidu.

Requirements

  • Create browser-emulator AMI:

    Before run the create an AMI, ensure that your AWS credentials are properly configured. You can check it running aws configure.

    Once aws credentials have been configured in your pc, it's time to create the browser-emulator AMI. For creating it, you only have to run the createAMI.sh script.

     cd ./browser-emulator/aws/
 ./createAMI.sh

    Defaults to creating the AMI:

    • In the us-east-1 AWS region.
    • Using Chrome as the browser for tests.
    • Not installing QoE analysis software required if doing QoE analysis in situ (generally not necessary).

    After AMI was created, you must set the AMI ID to load-test/src/main/resources/application.properties, in particular WORKER_AMI_ID property.

    You can change the default values of the script in the createAMI.sh script like this:

     cd ./browser-emulator/aws/
 ./createAMI.sh --region eu-west-1 --version ./EC2-browser-emulator-firefox.yml

    The following versions are available:

    • Chrome with no qoe analysis software preinstalled (default): ./EC2-browser-emulator-no-qoe.yml
    • Chrome with qoe analysis software preinstalled: ./EC2-browser-emulator.yml
    • Firefox with no qoe analysis software preinstalled: ./EC2-browser-emulator-firefox-no-qoe.yml
    • Firefox with qoe analysis software preinstalled: ./EC2-browser-emulator-firefox.yml

  • Create a security group:

    The instance which will contain the browser-emulator service must have the 5000 (REST API) and 5001 (WebSocket) ports opened. For doing this, you will have to create a AWS Security Group (SG) with these requeriments.

    After SG was created, you must set the security group ID to load-test/src/main/resources/application.properties, in particular WORKER_SECURITY_GROUP_ID property.

The loadtest-controller will use the BrowserEmulator AMI (previously created) for create instances on demand. All you have to do is fill the AWS configuration in loadtest-controller properties besides the required parameters

2. Execute controller

In the machine you want to execute the controller you need to have Java 11 platform installed.

Then follow these steps:

1. Clone this repository

git clone https://github.com/OpenVidu/openvidu-loadtest.git

2. Configure Loadtest Controller

Running the following command under project root directory you will be able to edit the load-test/src/main/resources/application.properties which contains a list of configuration parameters that you will can customize.

nano loadtest-controller/src/main/resources/application.properties

In this file you will see:

Required parameters

# Load Test Parameters (Required)
OPENVIDU_URL=https://openvidu_pro_url
OPENVIDU_SECRET=openvidu_pro_secret
SESSION_NAME_PREFIX=LoadTestSession
USER_NAME_PREFIX =User
SECONDS_TO_WAIT_BETWEEN_PARTICIPANTS=1
SECONDS_TO_WAIT_BETWEEN_SESSIONS=2
SECONDS_TO_WAIT_BEFORE_TEST_FINISHED=10
SECONDS_TO_WAIT_BETWEEN_TEST_CASES=10

Development mode parameters (only for testing locally)

# For testing locally, fill it with the worker ip address: 195.166.0.0
WORKER_URL_LIST=

AWS parameters (only for testing on AWS)

General AWS parameters

# For testing on AWS

# For launching EC2 instances and allowing to workers upload recordings to S3 bucket
AWS_ACCESS_KEY=your-aws-key
AWS_SECRET_ACCESS_KEY=your-secret-key
WORKER_AMI_ID=ami-xxxxxxxxxxx
# We recommend c5 type. https://aws.amazon.com/ec2/instance-types/
WORKER_INSTANCE_TYPE=c5.xlarge
# By default, the browser-emulator service is listening on:
# 5000 (REST API)
# 5001 (WebSocket)
# The SG will need these ports opened.
WORKER_SECURITY_GROUP_ID=sg-xxxxxxxxx
WORKER_INSTANCE_REGION=
# Numbers of workers to launch before the test starts
WORKERS_NUMBER_AT_THE_BEGINNING=1
For the AUTO distribution of participants to workers
# Overcoming this CPU percentage threshold a new worker will be launched
WORKER_MAX_LOAD=80
# Number of new workers incrementation
WORKERS_RAMP_UP=1
For the MANUAL distribution of participants to workers
# Boolean param for MANUAL participants allocation
MANUAL_PARTICIPANTS_ALLOCATION=false
# Number of participants per worker
USERS_PER_WORKER=
If you want to save the recordings in MinIO
################################################
#### MinIO credentials and information if using it to save recordings instead of S3
################################################
# MinIO Endpoint
MINIO_HOST=
# MinIO Port (default 443)
MINIO_PORT=443
# MinIO Access Key
MINIO_ACCESS_KEY=
# MinIO Secret Key
MINIO_SECRET_KEY=
# MinIO Bucket Name
MINIO_BUCKET=
For video quality control

These properties allow record a participant (launching an external chrome browser) for check the received video quality.

When test finishes, it will saved in the S3 Bucket.

# For video quality control (enabled for AWS testing only) 
# It'll start a new ec2 instance where a new participant will be connected using a real Chrome Browser
# it will start to record the session when media node has archieved this value
# Needs an ElasticSearch instance to work
MEDIANODE_LOAD_FOR_START_RECORDING=0
# it will start to record the sessions gruped by the specifyed value.
# 0: Recording disabled, 1 recording starts each session, 2 recording starts each two session ...
RECORDING_SESSION_GRUPED_BY=0
# Bucket name where the recordings will be saved
# Record each MediaStream in each worker for Quality of Experience analysis
QOE_ANALYSIS_RECORDINGS=true
# Perform qoe analysis in the same worker as they were recorded in, if false the recordings will only be uploaded to S3
QOE_ANALYSIS_IN_SITU=false
# Video information needed for in situ QoE analysis, read https://github.com/OpenVidu/openvidu-loadtest/tree/master/browser-emulator#running-qoe-analysis for more information
VIDEO_PADDING_DURATION=1
VIDEO_FRAGMENT_DURATION=5
S3_BUCKET_NAME=openvidu-loadtest-capacity

For retrying the participant creation

These properties allow retry the participant creation when it fails

RETRY_MODE=false
RETRY_TIMES=5

For selecting the video to use (only used in browserMode REAL)

These properties allow you to select the video to use when the browsers used are real browsers. There are 3 default options and you can also use a custom video, check the description of the browser-emulator Initialize instance API for more information about these options:

# Video type options: BUNNY, INTERVIEW, GAME, CUSTOM
VIDEO_TYPE=BUNNY
VIDEO_WIDTH=640
VIDEO_HEIGHT=480
VIDEO_FPS=30
# If VIDEO_TYPE is CUSTOM, you have to specify the video and audio URLs
VIDEO_URL=
AUDIO_URL=

Monitoring parameters

WARNING: The experiments we made have been tested with Elasticsearch 7.8.0 version. A deployment with a different version could cause errors.

# Monitoring Parameters (Optional)
ELASTICSEARCH_USERNAME=elasticusername
ELASTICSEARCH_PASSWORD=password
KIBANA_HOST=https://kibanahost

| Make sure your OpenVidu Configuration has the following properties for a good monitoring:

  • OPENVIDU_PRO_STATS_MONITORING_INTERVAL=1
  • OPENVIDU_PRO_STATS_WEBRTC_INTERVAL=1

Miscelaneous configuration

Recommended to leave default values unless you know what you are doing.

# Maximum number of concurrent requests. If left blank, defaults to number of available processors + 1.
MAX_REQUESTS=
# Install VNC server on workers for debug purposes
DEBUG_VNC=false

3. Configure session typology

Controller is being developed to allow the configuration of the load test cases: number of participants, typology, number of sessions, etc.

Currently, the only available option is the change of the number of participants of the session.

To configure the test cases the file loadtest-controller/src/main/resources/test_cases.json has to be edited:

{
	"testcases": [
		{
			"typology": "N:N",
			"participants": ["2", "3", "5"],
			"sessions": "20",
			"browserMode": "EMULATE",
			"frameRate": "30",
			"resolution": "640x480",
			"openviduRecordingMode": "",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) of publishers that the array of participants indicates"
		},
		{
			"typology": "N:M",
			"participants": ["1:10", "1:100", "2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
			"sessions": "infinite",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS and SUBSCRIBERS as the participants array indicates."
		},
		{
			"typology": "TEACHING",
			"participants": ["2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
			"sessions": "infinite",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS (teachers:students) as the participants array indicates. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio"
		},
		{
			"typology": "TERMINATE",
			"desciption": "This terminate all EC2 instances launched for loadtest purposes"
		}
	]
}
Test Case JSON properties
Properties Type Description
typology * N:N, N:M, TEACHING or TERMINATE N:N: All users will be PUBLISHERS and SUBSCRIBERS
N:M: N number will be PUBLISHERS and M number will be SUBSCRIBERS
TEACHING: It will emulate a teaching videoconference. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio
TERMINATE: It will terminate all EC2 instances launched for loadtest purposes
participants * String [] Number of participants in each session. It will run the same test case as many time as the positions in the array.
For example:
For N:N typology: ["2","5"] where all of them will be publishers and test case will be run twice (sessions with 2 and 5 participants)
For N:M typology: ["2:30"] where 2 will be publishers and 30 will be subscribers
For TEACHING typology: ["2:30"] where 2 will be teachers and 30 will be students
sessions * infinite or Number infinite: It will create infinite sessions until the limit is reached.
Number value: It will create only (number value) sessions
browserMode * EMULATE or REAL EMULATE: the service will emulate a browser.
REAL: the service will launch a Chrome browser docker container.
Choosing EMULATE, you must ensure that OpenVidu aren't forcing H264 coded
resolution String Desired resolution of the video.
640x480 or 1280x720
Default 640x480
frameRate Number (0-30) Desired framerate of the video in frames per second. Default 30
openviduRecordingMode String COMPOSED or INDIVIDUAL
See OpenVidu recording.
browserRecording Boolean If browserMode is REAL and you want record the Chrome browser using ffmpeg. Otherwise, If browserMode is EMULATE and you have started browser-emulator with KMS user type (see worker running options) Default false.
showBrowserVideoElements Boolean If browserMode is REAL and you want show videos elements into the app running in Chrome. Default true
headlessBrowser Boolean If browserMode is REAL and you want launch a headless Chrome. Default false. See Headless Chromium

3. Run load test

When you execute the load test the controller will create sessions and will connect participants into them automatically.

The load test has several stop conditions. It will stop when any of the following conditions occur:

  • When loadtest-controller receives an exception error from any of the workers.

  • When any of workers can't create a new participant because of lack of resources or a long time trying creating it.

Take into account that errors can be produced by OpenVidu errors or if the worker itself is overloaded. Please control CPU usage in workers.

When an error in a worker is produced, the load test will stop and will order to destroy and close every participants and sessions to the workers.

For start with it, run the following command:

cd loadtest-controller
mvn spring-boot:run

Extra features

  • For terminate AWS EC2 instances:
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-DTERMINATE_WORKERS=true"

Sample test execution

For illustration proporses here is an example composed by 2 workers and the load test app all of them running locally.

As you can see in the load test and workers logs, the load test app will create sessions and will add participants into them following a round-robin scheduling.

Load test logs Load Test Controller Logs

Which it means that, for this sample with sessions of 2 participans, each participant will be create by one different worker. The User0 will be create by the worker 1 and the User1 will be created by the worker2.

Thus achieving more capacity in the load test and less resource consumption.

Worker 1 logs

Worker 1 Logs

Worker 2 logs

Worker 2 Logs

Test results

The loadtest-controller will create a report result on the root directory /openvidu-loadtest/loadtest-controller with the name result.txt. This file will contains the following information:

 ----- Test Case Report Fri Apr 30 12:39:46 CEST 2021 -----
Browser approach:
Browser with recording:
Session typology:
Participants per session:
Number of sessions created:
Number of sessions completed:
Number of participants created:
Number of workers used:
Stop reason:
Test duration:
Kibana url:

If you're testing OpenVidu PRO, the loadtest-controller will import a Kibana Dashboard automatically at the beginning of the test. This dashboard will include Kibana Visualizations with all metrics retrieved from OpenVidu.

Load Test Dashboard

To allow that the Load Test App import it, you must fill the monitoring parameters.

What happen if the dashboard have not been imported automatically?

An alternative if the dashboard is not imported automatically is import it manually. You just to follow these steps:

  1. Go to your Kibana Home
  2. Open the toggle menu and go to Management > Stack Management.
  3. Once inside of Stack Management, you must click on Saved Objects option, under Kibana section.
  4. Here, you can find an import button. You have to clik on it and import the loadtest.ndjson file.

Besides, if you have deployed OpenVidu PRO you can create your own visualizations in Kibana or you can export the raw data and use another tool. Remember that ELK stack has monitoring information about the platform (CPU, memory usage, bandwidth, etc) and also high level information (sessions, participants, etc.)

Browser Emulator documentation

This service provides a simple REST API that will be used by Load Test application and it allows:

  • Ping to instance. Do ping to check if instance is ready.

  • Initialize instance. Initialize monitoring stuffs like ElasticSearch env variables and Metricbeat container. This request also set the AWS public and secret keys for uploading the video recordings files to S3 (just in case the test case includes recording).

  • Create a participant (PUBLISHER or SUBSCRIBER) using a custom token created by you or creating a new token.

  • Delete a specific participant by its connectionId

  • Delete all participant with a specific role (PUBLISHER or SUBSCRIBER).

  • Delete all participant

This services also is listening for a WebSocket communication on ws:browser-emulator-addres:5001/events. It will send information from openvidu-browser to the loadtest-controller.

See browser-emulator docummentation for more info.

Acknowledgments

This work has been supported by the Regional Government of Madrid (Spain) (CM) through project EDGEDATA-CM (P2018/TCS-4499) cofunded by FSE & FEDER.