Media2Cloud Solution
Media2Cloud solution is designed to demonstrate a serverless ingest framework that can quickly setup a baseline ingest workflow for placing video assets and associated metadata under management control of an AWS customer. The solution will setup the core building blocks that are common in a ingest strategy:
- Establish a storage policy that manages master materials as well as proxies generated by the ingest process.
- Provide a unique identifier (UUID) for each master video asset.
- Calculate and provide a MD5 checksum.
- Perform a technical metadata extract against the master video asset.
- Build standardized proxies for use in a media asset management solution.
- Run the proxies through the media analysis solution.
- Provide a serverless dashboard that allows a developer to setup and monitor the ingest process.
For more information and a detailed deployment guide visit the Media2Cloud Solution at https://aws.amazon.com/answers/media-entertainment/media2cloud/.
Running unit tests for customization
- Clone the repository, then make the desired code changes
- Next, run unit tests to make sure added customization passes the tests
cd ./deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh
Building distributable for customization
-
Configure the bucket name of your target Amazon S3 distribution bucket Note: You would have to create an S3 bucket with the prefix 'my-bucket-name-<aws_region>'; aws_region is where you are testing the customized solution. Also, the assets in bucket should be publicly accessible.
-
Now build the distributable:
chmod +x ./build-s3-dist.sh
./build-s3-dist.sh --bucket my-bucket-name-<aws_region> --version <my-version>
- Deploy the distributable to an Amazon S3 bucket in your account. Note: you must have the AWS Command Line Interface installed.
aws s3 cp ./dist/ s3://my-bucket-name-<aws_region>/media2cloud/<my-version>/ --recursive --acl bucket-owner-full-control --profile aws-cred-profile-name
- Get the link of the media2cloud-deploy.template uploaded to your Amazon S3 bucket.
- Deploy the Media2Cloud Solution to your account by launching a new AWS CloudFormation stack using the link of the media2cloud-deploy.template.
File Structure
The Media2Cloud Solution consists of a demo website, an ingest, metadata, and media-analysis orchestration layers, a search and storage layer, and an API layer.
- The demo website is a JQuery web application that interacts with Amazon S3, Amazon API Gateway, Amazon DynamoDB, AWS Iot Core, and Amazon Cognito.
- The ingest orchestration layer is an AWS Step Functions state machine that extracts mediainfo and creates proxy and thumbnail images of the uploaded file.
- The metadata orchestration layer is an AWS Step Functions state machine that invokes and monitors media-analysis sub-state machine. Upon analysis is completed, the metdata state machine processes the analysis metadata results and converts them into WebVTT tracks.
- The media-anlysis orchestration layer is an AWS Step Functions state machine that coordinates metadata extraction from Amazon AI services.
- The search and storage layer uses Amazon Elasticsearch to index extracted metadata and handle search requests.
- The API layer handles requests from the web client.
- The microservices are deployed to a serverless environment in AWS Lambda.
|-deployment/
|-buildspecs/ [ solutions builder pipeline build specifications ]
|-common.sh [ shell script shared by build-s3-dist and deploy-s3-dist scripts ]
|-build-s3-dist.sh [ shell script for packaging distribution assets ]
|-deploy-s3-dist.sh [ shell script to deploy solution to Amazon S3 bucket ]
|-run-unit-tests.sh [ shell script for executing unit tests ]
|-media2cloud-deploy.yaml [ solution CloudFormation deployment template ]
|-media2cloud-bucket.yaml [ solution CloudFormation template for deploying s3/glacier bucket ]
|-media2cloud-api-stack.yaml [ solution CloudFormation template for deploying API services ]
|-media2cloud-state-machine-stack.yaml [ solution CloudFormation template for deploying ingest and metadata state machines ]
|-media2cloud-webapp-stack.yaml [ solution CloudFormation template for deploying web bucket, cloudfront, and webapp ]
|-media-analysis-deploy.yaml [ solution CloudFormation template for deploying Media Analysis Solution ]
|-media-analysis-api-stack.yaml [ solution CloudFormation template for deploying Media Analysis API services ]
|-media-analysis-storage-stack.yaml [ solution CloudFormation template for deploying Media Analysis storage services ]
|-media-analysis-state-machine-stack.yaml [ solution CloudFormation template for deploying Media Analysis state machine ]
|-source/
|-backend/ [ microservices for orchestrating backend logics ]
|-index.js [ state machine lambda functions entrypoints ]
|-lib/
|-api/ [ microservice for handling requests from Amazon API Gateway ]
|-ingest/ [ microservice for ingest state machine ]
|-fixtures/ [ unit test fixtures ]
|-mediainfo/ [ microservice for extracting mediainfo from video file ]
|-postprocess/ [ microservice for handling ingest completion ]
|-preprocess/ [ microservice for handling s3 object arrival events ]
|-transcode/ [ microservice for create and monitor proxy file ]
|-metadata/ [ microservice for metadata state machine ]
|-analytics/ [ microservice for managing media-analysis state machine ]
|-mam/ [ microservice for sanitizing metadata results and dispatch message ]
|-preprocess/ [ microservice for copying proxy file to media-analysis specific location ]
|-vtt/ [ microservice for generating webvtt tracks ]
|-shared/ [ shared components for backend and frontend microservices ]
|-db.js [ helper class to Amazon DynamoDB service ]
|-dbConfig.js [ for accessing Amazon DynamoDB Configuration table ]
|-iotStatus.js [ helper class for publishing message to Iot core message broker ]
|-machineMetadata.js [ helper class to load vtt and metadata files ]
|-mxIngestAttributes.js [ mixer class to manage ingest attributes, used by GlacierAttributes class ]
|-mxCommonUtils.js [ mixer class to wrap common operations such upload, download, md5chksum ]
|-signer.js [ sigV4 from https://github.com/AnomalyInnovations/sigV4Client/blob/master/sigV4Client.js ]
|-stateIOData.js [ state machine input/output wrapper class ]
|-videoAsset.js [ class GlacierAttributes, ProxyAttributes, VideoAsset definitions ]
|-sns/ [ helper class to send message to sns topic ]
|-common.js [ export commonly use classes ]
|-custom-resources/ [ AWS CloudFormation custom resource for aiding the deployment of the solution ]
|-index.js [ lambda function entrypoint and routing to specific function ]
|-lib/
|-dynamodb/ [ helper to initialize Amazon DynamoDB Configuration table ]
|-iot/ [ helper to get AWS Iot endpoint and detach Iot policy onDelete ]
|-lambda/ [ helper to update environment variables, deprecated ]
|-mediaconvert/ [ helper to get AWS Elemental MediaConvert per region, per account endpoint ]
|-s3/ [ helper to set Amazon S3 CORS, Lifecycle, and lambda notifications ]
|-shared/ [ shared components ]
|-cfResponse.js [ wrapper to send response to AWS CloudFormation stack ]
|-mxBaseResponse.js [ mixer to provide unified CloudFormation response ]
|-sns/ [ helper to subscribe multiple email addresses to Amazon SNS topic ]
|-string/ [ helper to manipulate string, maxlen, minlen, upper, lower, underscore, dash ]
|-web/ [ helper to copy web assets to Amazon S3 bucket and create solution manifest JSON file ]
|-webapp/ [ demo web client for the solution ]
|-public/ [ static assets ]
|-src/
|-lib/
|-js/
|-app.js [ loader to UI components ]
|-appUtils.js [ helper class to provide authenticated http request ]
|-cardCollection.js [ main layout managing a collection of the video cards ]
|-fileUpload.js [ uploader modal dialog for managing file upload to Amazon S3 ]
|-iotSubscriber.js [ helper class to subscribe, receive, and delegate iot messages ]
|-myCognito.js [ wrapper class to Amazon Cognito for managing sign in/out, authenticate user ]
|-preview.js [ preview modal dialog ]
|-searchBox.js [ search box UI ]
|-settingsTabPanel.js [ setting tab UI ]
|-signInModal.js [ sign in modal dialog for user authentication ]
|-systemMessageArea.js [ text area to redirect console log/error messages ]
|-videoCard.js [ video card layout ]
|-third_party/
|-amazon-cognito-identity-bundle/ [ used at build time, from https://www.npmjs.com/package/amazon-cognito-identity-js ]
|-aws-iot-sdk-browser-bundel/ [ used at build time, from https://www.npmjs.com/package/aws-iot-device-sdk-browser ]
|-spark-md5-bundle/ [ used at build time, from https://www.npmjs.com/package/spark-md5 ]
Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
http://aws.amazon.com/asl/
or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.