UCT Opencast Build Script

Scripts used to build and deploy Opencast from source to multiple servers, each with their own configuration.

Getting Started

Make sure you have Java-8 JDK (not JRE) installed.

java -version javac -version
Make sure you have Maven (mvn) 3.3 or later installed, the java version should be the same as shown above (Building source).

mvn -version
Make sure you have Ansible 2.1 or later installed (Deploying to the respective servers).

ansible --version
Make sure you have XMLStarlet installed (To obtain the source build version).

apt-get install xmlstarlet
Check out oc-scripts into some folder.

git clone https://github.com/cilt-uct/oc-scripts.git /usr/local/src/scripts
Configure your config.sh file:

cp config-dist.sh config.sh

Deploy configurations

Remember to create and modify the appropriate deployment configuration file.

cp deploy-example.cfg deploy-dev.cfg
cp deploy-example.cfg deploy-prod.cfg
cp deploy-example.cfg deploy-staging.cfg
cp users-example.cfg users.cfg

cp hosts/example hosts/dev
cp hosts/example hosts/production
cp hosts/example hosts/staging

Configuring Servers

Add the server name to the appropriate hosts file 'hosts/'

dev: Development servers
production: Production servers
staging: Staging servers

Create a folder in files/config with the name of the server e.g srvubuopc001 and the following structure

files/config/[servername]
├── bin
└── etc

Configure server build file:

cp build-example.cfg build-[servername].cfg

Note: additional configuration parameters is defined in build-all.cfg.

Configure server configuration file (used when the distribution is deployed on the server):

cp conf-example.cfg conf-[servername].cfg

Note: server_url in conf-[servername].cfg should be identical to deploy_server_name in build-[servername].cfg.

Running

Usage: run.sh [options] (Deploy type: dev, production | prod, staging)

Example: run.sh -bd prod Build source, package and deploy to production servers.

NOTE: If configured some options also add JIRA comments and New Relic deployment markers.

Options:

-h, --help This help text.

-a, --all Update the source, build it, clean the servers and deploy new build.

--alias Deploy the Opencast bash aliases file to each server to make interaction with the sytem easier.

-b, --build Will build the assemblies for $SRC (e.g mvn clean install) with the current source.

-c, --clean Will clean the deployment of opencast on all the associated servers. a) Clean Database b) Clean Indexes, Distribution and Workspace

-d, --deploy Deploy the currently build assemblies to their respective servers.

-t, --lti-deploy Deploy LTI tool static files to the appropriate servers.

-l, --list List the servers that can be updated with this script.

--ocr Deploy the Tesseract and hunspell data and dictionary files.

--track4k Deploy Track4k binary files to the appropriate nodes.

--audiotrim Deploy or update the audio trim detector script from github.

--emptyvenue Deploy or update the empty venue detector script from github.

-r, --reconfig Reconfigure the respective servers. Deploy configuration build to each, which includes custom.properties, encoding profiles and workflows.

-u, --update-git Will update $SRC to include the latest changes on the selected branch. e.g git fetch && git pull

-s, --status Display the current git status of the source folder ($SRC) and assemblies.

-v, --version Run a script on each server and return the version of all the program dependencies.

-x, --xtop Stop all opencast services on the respective servers.

-z, --ztart Start all opencast services on the respective servers.

-- Do not interpret any more arguments as options.

Note: The deploy and reconfiguration on production and staging will not be run if the source or script folder are not up-to-date with respects to their repositories (all changes committed).

Keeping it all up-to-date

It might be good practice to setup the repository to use access keys (SSH keys / read-only) to keep it up-to-date (see Github repository Settings > Access keys for more information).

To update repository.

./get.sh
To push any commits back to master.

/push.sh

After Deploy

Reconstruct the Admin UI search index. There are two ways to reconstruct the index:

By opening http://mediadev.uct.ac.za/admin-ng/index/recreateIndex or http://media.uct.ac.za/admin-ng/index/recreateIndex in your browser. The resulting page is empty but should return an HTTP status 200 (OK).

By using the REST documentation, open "Admin UI - Index Endpoint" and use the testing form on /recreateIndex. The resulting page is empty but should return an HTTP status 200 (OK). You can find the REST documentation in the help-section of the Admin UI behind the ?-symbol.

TODO: Add and review https://docs.opencast.org/latest/admin/upgrade/.

Workflows

ID	Title	Description	UCT	Tag
partial-cleanup	Cleanup after processing		❌	-
partial-watermark	Render watermark into presenter and presentation tracks		❌	-
uct-clean	UCT - Retract and Cleanup	Retract a recording and clean out media files.	✔️	archive
uct-detect-empty-venue	UCT - Empty venue detection	Run VAD script, to analyse speech segments and in turn identify if venue is empty.
uct-discard	UCT - Discard Media	Discard all the media from this event.	✔️	archive
uct-include-autotrim-detection	Audio trim point detection	Run the audio analysis for (possible) autotrimming of the recording	✔️	archive
uct-include-before-edit-auto-trim	UCT - Process for Editing (Auto Trim)	Publish directly from ingest process using autodetected trimpoints.	✔️	-
uct-include-before-edit-review	UCT - Process for Editing (Review)	Using the work flavors this workflow creates */preview versions for review.	✔️	-
uct-include-partial-download-abde	Download for ABDE		✔️	-
uct-include-partial-preview-abde	Preview for ABDE		✔️	-
uct-include-transcription-watson	Submit trimmed audio for Watson Transcription	Runs after real publishing of media	✔️	-
uct-include-transcription-nibity	Submit trimmed audio for Nibity Transcription	Runs after real publishing of media	✔️	-
uct-ingest-only	UCT - Ingest Only	Ingest source material	✔️	schedule
uct-partial-ingest	UCT - Ingest asset		✔️	-
uct-partial-preview	Prepare preview artifacts (/work > /preview)		✔️	-
uct-partial-publish-coverimage	UCT - Publish Coverimage		✔️	x
uct-partial-publish-downloads	UCT - Publish Downloads		✔️	-
uct-partial-publish	Publish the recording		✔️	-
uct-partial-work	Prepare work versions		✔️	-
uct-partial-work-channel	Prepare work versions - with selection of audio channel		✔️	-
uct-process-before-edit	UCT - Process for Editing		✔️	upload, schedule
uct-process-obs	UCT - Process OBS Recording		✔️	schedule
uct-process-personal	UCT - Process for Personal Series		✔️	schedule
uct-process-track4k	UCT - Process Source with Track4K		✔️	archive
uct-process-upload	UCT - Process Upload		✔️	upload
uct-publish-after-edit	UCT - Publish		✔️	editor
uct-publish-feed	UCT - Publish feed		✔️	-
uct-request-consent	Request consent		✔️	-
uct-test	Testing workflow	Various	✔️	various
uct-transcript-watson	Submit trimmed audio for Watson Transcription		✔️	archive
uct-transcript-nibity	Submit trimmed audio for Nibity Transcription		✔️	archive
uct-update-previews	Update previews		✔️	archive

cilt-uct/oc-scripts