This solution provides a reference architecture and implementation steps to build a platform either for short form videos for a social app, or long form videos for a streaming service on AWS which is highly available and delivers video content with low latency in a secured way.
The solution includes a video processing pipeline to store, process and deliver the videos to the end user along with a serverless API layer to handle the dynamic calls from client-side application. The solution also provides a payment gateway integration to manage payments.
Figure 1. Media Streaming Platform Architecture
The architecture includes the following components:
- Amazon S3 Buckets (source and destination) to store the uploaded and processed videos.
- AWS Elemental Mediaconvert to transcode the videos into HLS adaptive bitrate files.
- Amazon Cloudwatch to track the mediaconvert job and invoke the required lambda function.
- Amazon Cloudfront to distribute the video content.
- Amazon API Gateway to route the incoming HTTP requests from client side to Lambda function.
- AWS Lambda functions to handle the triggers for video processing and HTTP requests from API Gateway.
- Amazon DynamoDB tables to store the user and video metadata.
- Amazon Opensearch Service to provide real-time search.
- AWS Amplify for hosting client side web application and managing authentication using Amazon Cognito.
- Razorpay for managing payments. (Razorpay service is provided by third party vendor)
AWS Identity and Access Management (IAM) provides fine-grained access control to the services in the AWS cloud. The solution creates multiple IAM roles to allow access to specific services to achieve least-privilege permission.
Amazon Cognito adds the authentication and authorization to the client-side application. The end users need to authenticate themselves to upload or access the available content.
To help reduce latency and improve security, this solution includes an Amazon CloudFront distribution with an origin access identity, which is a special CloudFront feature that helps restrict access to the solution's website bucket contents.
Before you launch the automated deployment, review the architecture discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the Media Streaming Platform solution into your account.
You are responsible for the cost of the AWS services used while running this solution on your AWS account.
- Download the automated Cloudformation template, Lambda functions and the source code for client side application which is available on the Github Repository.
AWS Employees having Workdocs access can download the code from here.
This repository contains the source code for the following assets:
- AWS Cloudformation template ( CFTemplates/media-streaming- platform.json ) to deploy the AWS resources in your AWS account.
- AWS Lambda functions( LambdaFunctions ) for client side application to interact with the AWS resources.
- An implementation of a client side application to host the Media Streaming Platform which can be used as a reference to build application as per your requirements.
- IAM Policy ( IAM-Policies/Cognito-Auth-Role.json ) to provide access to S3 bucket to upload files to S3.
- Download zip files for all the Lambda Functions ( LambdaFunctions ) from Github repository and upload them to an Amazon S3 Bucket ( vod-lambda-functions ) in your AWS account. You can create a new Amazon S3 Bucket to store the Lambda functions by following the steps mentioned here.
The automated AWS CloudFormation template deploys the components of the Media Streaming platform on the AWS Cloud.
The template deploys a number of resources. Therefore you must select the AWS region in which all the services are available. You can refer toAWS Regional Service List.
- Download the Cloudformation template.
- Go to AWS Console and open Cloudformation.
- On the Create stack page, verify that the correct template shows in the Amazon S3 URL text box and choose Next.
- On the Specify stack details page, assign a name to your solution stack.
- Under Parameters , review the parameters for the template, and modify them as necessary. This solution uses the following default values.
| Parameter Name | Default Value | Description |
|---|---|---|
| lambdaFunctionsS3Bucket | vod-lambda-functions | S3 Bucket Name in which the Lambda functions are stored |
| region | us-east-1 | AWS region in which the resources are launched |
- Choose Next.
- On the Options page, choose Next.
- On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.
- Choose Create to deploy the stack. You can view the status of the stack in the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in approximately five minutes.
- The values in the output tabs would be required in configuring the client-side application.
- Go to AWS Console and Open Lambda.
- Select the opensearchIndexing function created by cloudformation.
- Select Configuration.
- Select Environment Variables.
- Update the OPENSEARCH_DOMAIN value from the Outputs tab of Cloudformation Stack.
- Follow steps 12-15 for opensearchFetch function.
This section can be skipped if you do not want to integrate a payment gateway using Razorpay in client-side application.
Razorpay is used to integrate the payment gateway in the client-side application. You need to visitRazorpay to generate the API keys.
Follow these steps to generate API keys:
- Log into yourDashboard with appropriate credentials.
- Select the mode ( Test or Live ) for which you want to generate the API key.
- Test Mode : The test mode is a simulation mode that you can use to test your integration flow. Your customers will not be able to make payments in this mode.
- Live Mode : When your integration is complete, switch to live mode and generate live mode API keys. Replace test mode keys with live mode keys in the integration to accept payments from customers.
- Navigate to Settings → API Keys → Generate Key to generate keys for the selected mode. The Key Id and Key Secret appear on a pop-up page.
- Go to AWS Console and open Lambda in AWS Console.
- Select createPaymentOrder function.
- Select Configuration
- Select Environment Variables
- Update the following environment variables with the Razorpay Key Id and Razorpay Key Secret generated in step 3. A. RAZORPAY_KEY_ID = <RAZORPAY_KEY_ID> B. RAZORPAY_KEY_SECRET = <RAZORPAY_KEY_SECRET>
- CloneGithub Repository. git clone https://github.com/knight-18/video-on-demand.git
- Enter into the root directory cd msp
- Install the required modules for client-side application npm install
- To initialize a new Amplify project, run the following command from the root directory. amplify init
- Enter name for the project
- Setup basic configuration and authentication method
- Select AWS region
- To add authentication services, run the following command amplify add auth
- Select default configuration
- Select email for signin method.
- Select No I'm done for advanced settings. (Amazon Cognito provides a number of features to provide more security to your application. You can also configure advanced security methods according to your requirements.)
- To update the cloud resources amplify push
- Create a file .env in the root directory. MacOS, linux users can use the following command the create an empty file: touch .env
- Copy contents of .env.example to .env.
- Copy the values from the output tab of Cloudformation stack and update in the .env file.
- The REACT_APP_COGNITO_IDENTITY_POOL_ID variable can be found in the file aws-exports.js as _aws_cognito_identity_pool_id_inside src folder.
- Go to AWS Console and open Amazon Cognito.
- Select the user pool created by the cloudformation template.
- Go to user pool properties.
- Click add Lambda Trigger.
- Select Pre-Signup Trigger.
- Select onCognitoUserCreate function.
When you create an Identity Pool, two roles will be created for you by Cognito. One for "Unauthenticated Role" and another for "Authenticated Role" as shown in the image above. We will need to modify the Authenticated Role roles using the IAM console. We will edit the roles using the policy editor and use the policy json below.
-
Go to AWS Console and open IAM.
-
Select Roles.
-
Search for "authRole" created by Amplify.
-
Select Add Permission.
-
Select Create Inline Policy
-
Select JSON Tab. Copy the downloaded policy ( IAM-Policies/Cognito-Auth-Role.json ) from Github Repository and replace the bucket name (my-test-bucket-amplify) with the name of the SourceBucket resource generated in Cloudformation outputs.
-
Select Review Policy.
-
Enter name (Source-Bucket-Access-Policy) and select Create Policy.
For running application on you local machine, use the following command from the root directory:
npm start
You need to create(if you already have an account you can use that) andsetup a github account on your local machine before using for the following steps.
-
Create a github repository and push your code.
-
Sign in to GitHub and create a new empty repository page.
-
Give a name (Media-Streaming-Platform) to the repository and select the public repository type. Leave other things to default and select Create Repository.
-
Go to the root directory of the client-side application in your terminal on your local machine.
-
Add the files to Git index git add .
-
Commit Added Files git commit -m "Added my project"
-
Add new remote origin (in this case, GitHub) _git remote add origin git@github.com:_username/Media-Streaming-Platform.git Replace the highlighted bits above with your username and repository name.
-
Push to GitHub git push -u origin master
-
Go to AWS Console and open AWS Amplify.
-
Select the deployed project.
-
Select "Hosting environments".
-
Select "Github" and click "connect branch" to allow access to your repository.
-
Select Media-Streaming-Platform repository from the dropdown list.
-
Select branch "master" and click "next".
-
Select an environment for the application.
-
Select role or create new if it doesn't exist already.
-
To create a new role, click create new role.
-
Select next:Permissions.
-
Select next:Tags.
-
Review the role and select create role.
-
Click on "advanced settings" and add all environment variables added in the .env file.
-
Click "next".
-
Review the details and click save and deploy.
- usersTable - Table to store users data
- shortVideosTable - Table to store short video data
- longVideosTable - Table to store long video data
- shortVideoLikesTable - Table to store short video likes data
- genrewiseVideosTable - Table to store long videos data with mapping to genre
- languagewiseVideosTable - Table to store long videos data with mapping to language
- longVideosRatingTable - Table to store long videos rating data
- subscriptionTable - Table to store users subscription status information
- onCognitoUserCreate - Updates user info to usersTable when new user sign up
- onShortVideoUpload - Updates short video data in shortVIdeosTable on short video upload
- onLongVideoUpload - Updates long video data in longVIdeosTable, genrewiseVideosTable and languagewiseVideosTable
- getShortContent - Fetch short videos from database
- getLongContent - Fetch long videos from database
- likeShortVideos - Function to like short videos
- rateLongVideo - Function to rate long videos
- queryDB - function to query database tables
- createPaymentOrder - Function to create payment order for Razorpay
- premiumSubscription - Function to update subscription details in database
- CustomResource - Function to manage video processing with AWS Elemental MediaConvert
- JobSubmit - Function submits the job to AWS Elemental MediaConvert
- JobComplete - Function to update video information after getting processed
- HTTP API - HTTP API to call lambda function from front end application
- uploadShortContentRoute - PUT /short-content/upload - Route to upload short videos
- uploadLongContentRoute - PUT /long-content/upload - Route to upload long videos
- getShortContentRoute - POST /short-content/get - Route to fetch short videos
- getLongContentRoute - POST /long-content/get - Route to fetch long videos
- likeShortContentRoute - POST /short-content/like - Route to like short videos
- rateLongContentRoute - POST /long-content/rating - Route to rate long videos
- dbQueryRoute - POST /query - Route to query database
- createPaymentOrderRoute - POST /premium/payment - Route to create payment order
- premiumSubscriptionRoute - POST /premium/confirm - Route to update subscription details
- opensearchFetchRoute - GET /opensearch - Route to fetch search results
- opensearchIndexingRoute - POST /opensearch - Route to index video in opensearch
- Source Bucket - Bucket to store uploaded videos by users
- Destination Bucket - Bucket to store videos converted into multiple formats
- Cloudfront Logging Bucket
- CloudFrontCloudFrontDistribution824F3346 - Cloudfront distribution to server content from destination bucket
- MediaConvertRole - MediaConvert transcodes the video into HLS Adaptive Bitrate files.
- Opensearch Domain - OpenSearch enables you to easily ingest, secure, search, aggregate, view, and analyze data.