/aws-scalable-big-blue-button-example

Demonstration of how to deploy a scalable video conference solution based on Big Blue Button

Primary LanguageShellMIT No AttributionMIT-0

Status: Work-in-progress. Please create issues or pull requests if you have ideas for improvement.

Scalable Big Blue Button Video Conference Example

Example deployment of a scalable Video Conference System

Summary

This project demonstrates the deployment of a scalable Video Conference Setup jointly using open-source software components and AWS Services.

High-Level Architecture

Architecture

Disclaimer

This project is an example of an deployment and meant to be used for testing and learning purposes only. Do not use in production.

Please note that running this code will cause software that is licensed under AGPL-3.0 to be deployed in the user’s account.

Be aware that the deployment is not covered by the AWS free tier. Please use the AWS pricing calculator to an estimation beforehand

Table of Contents

  1. Getting started
  2. Prerequisites
  3. Parameters
  4. Templates
  5. Todo
  6. Resources
  7. Security
  8. License

Getting started

Just a few steps are needed to get started with the example deployment. the deployment process is separated in a prerequisites deployment containing the creation of the source file Amazon S3 Bucket and another containing the actual deployment of the infrastructure and application layer.

You may use the included setup script to simplify and automatic deployment or alternatively you can run the deployment step-by-step.

Prerequisites

To run the automated, setup script based deployment you need to have some software installed and configured on your device:

To run the step-by-step setup:

either way you choose the following have to be in place:

Parameters

Dynamic parameters

These parameters you have to pass to the setup script

Parameter Name Value
-p the aws-cli profile to use
-e the operators email address
-h the hosted zone ID the DNS records to be added
-s the Cloudformation stack name you want to use
-d the FQDN for (aligned to the hosted zone )

Deployment parameters:

The deployment parameters are placed into the bbb-on-aws-param.json or to be set via cli/console ( if you choose the step-by-step setup. )

Parameter Name Default Value Description Comment
BBBECSInstanceType t3a.large Instance size of the ECS Cluster worker nodes should be aligned with the size VCPU and Memory limits of the to be deployed tasks below
BBBApplicationInstanceType t3a.xlarge Instance size of the Big Blue Button Application node(s) please refer to the Big Blue Button Documentation for rightsizing
BBBApplicationDataVolumeSize 20 the size of the application data volume used for recording buffer
BBBTurnInstanceType t3.micro Instance size of the turn server For right sizing please refer to the Big Blue Button Documentation
BBBDBInstanceType db.t3.medium Instance size of the Aurora Database Instance Heavily related to usage, collect metrics and test.
BBBCACHEDBInstanceType cache.t3.micro Instance size of the Redis security token and call ID handling Depends on usage.
BBBVPCcidr 10.1.0.0/16 The Cidr block for the VPC created during the deployment we deploy an own VPC for the deployment containing public and private subnets as well nas internet and nat gateways
BBBPrivateApplicationSubnets 10.1.5.0/24,10.1.6.0/24,10.1.7.0/24 The cidr blocks of subnets to be created within the VPC for the non-public components of the application deployment have to be 3
BBBPrivateDBSubnets 10.1.9.0/24,10.1.10.0/24,10.1.11.0/24 The cidr blocks of subnets to be created within the VPC for the database backend. have to be 3
BBBPublicApplicationSubnets 10.1.15.0/24,10.1.16.0/24,10.1.17.0/24 The cidr blocks of subnets to be created with the VPC for the direct public accessible application components have to be 3
BBBECSMaxInstances 10 The maximum amount of instances the ECS cluster should scale out to set a reasonable maximum to prevent cost explosion on unexpected usage
BBBECSMinInstances 1 The minimum amount of worker instances at the ECS cluster
BBBECSDesiredInstances 3 The desired amount of instances of worker instances at the ECS cluster
BBBApplicationMaxInstances 1 The maximum amount of Big Blue Button Application servers Set depending on the awaited load and planned instance size.
BBBApplicationMinInstances 1 The minimum amount of Big Blue Button Application servers As EC2 Autoscaling is currently not aware of ongoing video conferences, i recommend set min=max=desired and not use dynamic here (planned scale out/in)
BBBApplicationDesiredInstances 1 The desired amount of Big Blue Button Application servers As EC2 Autoscaling is currently not aware of ongoing video conferences, i recommend set min=max=desired and not use dynamic here (planned scale out/in)
BBBTurnMaxInstances 1 The maximum amount of Turn Servers to be deployed at the current state leave it to 1. Multiple Turn servers will need additional configuration on the turnserver
BBBTurnMinInstances 1 The minimum amount of Turn Servers to be deployed at the current state leave it to 1. Multiple Turn servers will need additional configuration on the turnserver
BBBTurnDesiredInstances 1 The desired amount of Turn Servers to be deployed at the current state leave it to 1. Multiple Turn servers will need additional configuration on the turnserver
BBBDBName frontendapp Set a Database Name for Greenlight / Scalelite
BBBDBEngineVersion 10.7 Set the Postgres version to be used at the Amazon Aurora setup please refer to the Amazon Aurora documentation for supported versions
BBBEnvironmentStage dev can be set to "dev","stage" or "prod" currently stage or prod does change the Amazon Aurora Setup to a Multi-AZ Setup and adds a 2nd Nat-Gateway to the deployment.
BBBEnvironmentName bbbonaws the name of the environment
BBBEnvironmentType scalable can be either "scalable" or "single" scalable for full scalable deployments. Single does leave out the ECS cluster, scalelite inner-application load balancer and Databases and installs Big Blue Button and Greenlight on a single EC2 instance and a turn server instance.
BBBgreenlightImage bigbluebutton/greenlight:v2 greenlight container image to be used
BBBScaleliteApiImage blindsidenetwks/scalelite:v1-api scalelite api container image to be used
BBBScaleliteNginxImage blindsidenetwks/scalelite:v1-nginx scalelite nginx container image to be used
BBBScalelitePollerImage blindsidenetwks/scalelite:v1-poller scalelite poller container image to be used
BBBScaleliteImporterImage blindsidenetwks/scalelite:v1-recording-importer scalelite recording importer container image to be used
BBBCacheAZMode cross-az Deploy the Redis cluster cross-az or single-az only cross-az supported atm
BBBGreenlightContainerMemory 1024 memory limit of the Greenlight container
BBBGreenlightContainerCPU 512 vCPU limit of the Greenlight container
BBBScaleliteImporterContainerMemory 256 memory limit of the Scalelite recording importer container usually does not need much resources, watch the metrics
BBBScaleliteImporterContainerCPU 128 vCPU limit of the Scalelite recording importer container usually does not need much resources, watch the metrics
BBBScalelitePollerContainerMemory 256 memory limit of the Scalelite application server poller container usually does not need much resources, watch the metrics
BBBScalelitePollerContainerCPU 128 vCPU limit of the Scalelite application server poller container usually does not need much resources, watch the metrics
BBBScaleliteNginxContainerMemory 512 memory limit of the Nginx Reverse Proxy container check metrics for sizing
BBBScaleliteNginxContainerCPU 512 vCPU limit of the Nginx Reverse Proxy container check metrics for sizing
BBBScaleliteApiContainerMemory 1024 memory limit of the Scalelite API container check metrics for sizing
BBBScaleliteApiContainerCPU 512 vCPU limit of the Scalelite API container check metrics for sizing
BBBScaleliteAddServerContainerMemory 512 memory limit of the Greenlight container does not need many resources
BBBScaleliteAddServerContainerCPU 128 vCPU limit of the Greenlight container does not need many resources
BBBSesRegion - Region of the SES Service to be used if the setup is planned to be deployed in a Region w/o Amazon SES, choose a proper region here.

Deployment

Automatic

For the automatic deployment just run the included setup script

Example:

./setup.sh -e johndoe@example.com -p bbb_example -h 1XXX02XXXXBMXXXXXZXXXX -s bbbexample -d bbbexample.example.com

The automatic deployment works as follows:

  • The setup script will validate the device prerequisites are met and all needed parameters are set.
  • It will then validate the syntax of the Amazon Cloudformation templates prior to execute any deployment.
  • It's going to deploy the Amazon S3 Bucket needed by the main deployment and read out the Bucket name as well as the name of the Stack deployed.
  • It will copy the needed scripts, config files for application and services as well as nested templates to the the deployed Bucket.
  • The main deployment will be executed. The script will read the content of the bbb-on-aws-param.json file and pass it through the stack deployment

Step-by-step

If you want to attempt the deployment step-by-step via Console or aws-cli please use the following steps:

  • deploy the Source Amazon S3 Bucket for scripts, config files and nested templates
aws cloudformation deploy --stack-name bbbexample-sources --profile=bbb_example --template ./templates/bbb-on-aws-buildbuckets.template.yaml

    aws s3 sync --profile=bbb_example ./templates s3://NAMEOFCREATEDBUCKET
    aws s3 sync --profile=bbb_example ./scripts s3://NAMEOFCREATEDBUCKET
  • start the deployment using the stackname of the stack deployed beforehand as one of the parameters:

using aws-cli:

aws cloudformation deploy --profile=bbb_example --stack-name bbbexample \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides BBBOperatorEMail=johndoe@example.com BBBStackBucketStack=bbbexample-sources BBBDomainName=bbbexample.example.com BBBHostedZone=1XXX02XXXXBMXXXXXZXXXX \
    --template ./bbb-on-aws-master.template.yaml

The deployment will take approx 30-45 minutes.

Logging into the Big Blue Button:

When deployment went through you find the Administrator login within Secretsmanager. The password is located at the secret starting with BBBAdministratorlogin- While your login has been set to your selected Operator Email Address the password has been generated for you.

log into the Frontend using conference.example.com

Template structure and deployment workflow

The Deployment consists of 2 main templates and 13 nested templates.

Main templates

  • The deployment of prerequisites via bbb-on-aws-buildbuckets.template.yaml

    The template deploys the Amazon S3 Bucket containing the scripts used at the turn, application and scalelite deployment as well as the nested templates source files.

  • The Master Template for the main deployment bbb-on-aws-master.template.yaml

    The template initiates the overall deployment of the scaling Big Blue Button example deployment.

Nested templates

  • Providing the Certificate Automation via: bbb-on-aws-acmcert.template.yaml

    This template deploys a custom resource into your AWS Account which provides full automation of requesting and validating Amazon Certificate manager based TLS Certificates. The validation will be done via Route53 DNS records.

  • Dynamically provide the latest AMI to be used with: bbb-on-aws-amifinder.template.yaml

    The template deploys a custom resource into your AWS Account which will access the AWS API to find the ami of the desired non-Amazon Linux Operating System.

  • Setup Email sending via Amazon SES: bbb-on-aws-ses.template.yaml

    This template deploys a custom resource which fully automates the Amazon SES configuration for sending mails and validating the domains in Amazon SES via Route53 records.

  • Building the network infrastructure using: bbb-on-aws-network.template.yaml

    The template deploys a full Amazon VPC with public and private subnets, route tables, internet gateways, nat gateways.

  • Provisioning of the Shared Storage: bbb-on-aws-storage.template.yaml

    This template deploys an Amazon Elastic File System into the setup to provide shared storage for the video conference recordings

  • Deploy Amazon Aurora (Postgres): bbb-on-aws-database.template.yaml

    The deployment of Amazon Aurora is needed to provide a database for Greenlight and Scalelite where the video conference schedules, user data and recording information are persistent

  • Deploy Amazon Elasticache (Redis): bbb-on-aws-cachedb.template.yaml

    This template deploys an Amazon Elasticache (redis) cluster where security token and conference IDs are located for the call handling via Scalelite

  • Fire up the ECS Cluster: bbb-on-aws-ecs.template.yaml

    The template deploys the ECS cluster and EC2 Autoscaling Group with the Launch Configuration for the Amazon EC2 worker nodes.

  • Add a turnserver to the stack: bbb-on-aws-bbbturn.template.yaml

    We need a turn server to handle the video call if the attendees are located behind a restrictive firewall

  • Initiate the frontend and inner-application load balancer to the ECS Cluster: bbb-on-aws-frontendapps.template.yaml

    Greenlight is providing the interface for the Users and the video conference landing page. Scalelite is the inner-application load balancer which provides conference load balancing over multiple big blue button application servers

Custom scripts

During the deployment the EC2 instances will be bootstrapped using UserData. To orchestrate the turn and Big Blue Button servers we will deploy some custom scripts helping to glue the components of the system

  • route53-handler systemd service and script deployed to the application and turn server instances creates dynamic records in your hosted zone (Route53) on boot/bootstrap of an instance and removes the record from the hosted zone.
    We do not use ElasticIPs or fixed hostnames at the setup to prevent e.g. service quotas. Also we use dynamic hostnames for each bootstrapped instance to prevent e.g. Let's encrypt penalties
  • scalelite-handler systemd service and script *deployed to each application instance adds the instance to scalelite on boot/bootstrap as active video conference instance and evaluates/removes the instance on shutdown"
  • turn-handler systemd service/timer and script the turn-handler the currently active turnserver hostname can be found by the application instances. As we dynamically set hostnames the turn servername might change on scaling or termination events. the turn-handler is started via systemd timer every X seconds
  • scalelite post script and config the script and config files are added to the Big Blue Button application instances to enable the recordings import into scalelite and Amazon EFS

Configuration adjustments (diffs from the defaults)

  • Cloudwatch Agent for application and turn instances the agent is automatically setup via UserData on bootstrap. To sent valid data to the Amazon Cloudwatch Service a custom agent config is used for application and turn servers

Resources

Security

See CONTRIBUTING for more information.

License

This Example is licensed under the MIT-0 License. See the LICENSE file.

3rd party Licensing

Please be aware of the deviating licenses of the deployed open-source software components.

Please note that running this code will cause software that is licensed under AGPL-3.0 to be deployed in the user’s account.