damato-ci
CI
Setup and config Jenkins
Install approaches
Requirements:
- OS: Ubuntu 16.04 or newer
Setup using docker on Ubuntu system:
sudo bash jenkins_install_docker.sh
Standardize Pipeline
6 stages:
- Prepare build environment
- Clone/Pull project from scm url
- Read
cli-config.json
in project source tree - Create docker volume for job
- Pull/Update ci tools docker image
- Pull/Update base docker image
- Download model to job's volume
- Download dataset to job's volume
- Prepare run environment inside base image
- Run evaluation command
- Save report
Developer Steps
To create job:
- In source code, specify
cli-config.json
file, for ex:
{
"model": {
"version": "0.0.1",
"dataset_ids": ["0123", "1b34", "333"],
"model_file_ids": ["1mD_MAD4SSXjuF0pPapaj3khxX6wICZwh", "1BOgYnnVcCvwoB6L8F0CW2aBb151TsHXr"]
},
"build": {
"base_image": "bachngocson/python3.6",
"environment_preparation": [
"cat /etc/os-release && env && ls -la",
"ls -la /root && ls -la /root/project_source_code",
"pip3 install -r requirements.txt"
],
"main": "python model.py"
}
}
-
In Web UI, specify folling information (* means required):
- * Jenkins server url
- * Job name
- * Github url
- Github Display name
- Pipeline library
- Pipeline version
- Pipeline template
The Web Server then calls bash script
cli_client_create_job.sh
with above parameters to create job and trigger first build -
In Web UI, if the user need to re run the build, specify folling information (* means required):
- * Jenkins server url
- * Job name
The Web Server then calls bash script
cli_client_build_job.sh
with above parameters to trigger a new build -
In Project's Github Settings page, add webhook URL:
Jenkins_server_url/github-webhook/
to trigger build whenever developer push code to Github. -
In Pipeline's Github Settings page, add webhook URL:
Jenkins_server_url/github-webhook/
to trigger build whenever the pipeline definition changes
Google Drive interaction
We'll use gdrive for interacting with Google Drive.
gdrive usage:
gdrive [global] list [options] List files
gdrive [global] download [options] <fileId> Download file or directory
gdrive [global] download query [options] <query> Download all files and directories matching query
gdrive [global] upload [options] <path> Upload file or directory
gdrive [global] upload - [options] <name> Upload file from stdin
gdrive [global] update [options] <fileId> <path> Update file, this creates a new revision of the file
gdrive [global] info [options] <fileId> Show file info
gdrive [global] mkdir [options] <name> Create directory
gdrive [global] share [options] <fileId> Share file or directory
gdrive [global] share list <fileId> List files permissions
gdrive [global] share revoke <fileId> <permissionId> Revoke permission
gdrive [global] delete [options] <fileId> Delete file or directory
gdrive [global] sync list [options] List all syncable directories on drive
gdrive [global] sync content [options] <fileId> List content of syncable directory
gdrive [global] sync download [options] <fileId> <path> Sync drive directory to local directory
gdrive [global] sync upload [options] <path> <fileId> Sync local directory to drive
gdrive [global] changes [options] List file changes
gdrive [global] revision list [options] <fileId> List file revisions
gdrive [global] revision download [options] <fileId> <revId> Download revision
gdrive [global] revision delete <fileId> <revId> Delete file revision
gdrive [global] import [options] <path> Upload and convert file to a google document, see 'about import' for available conversions
gdrive [global] export [options] <fileId> Export a google document
gdrive [global] about [options] Google drive metadata, quota usage
gdrive [global] about import Show supported import formats
gdrive [global] about export Show supported export formats
gdrive version Print application version
gdrive help Print help
gdrive help <command> Print command help
gdrive help <command> <subcommand> Print subcommand help
So, to download model, we use sh "gdrive-linux-x64 download -r -f --path /data/model/ ${model_file_id}"
in Jenkins pipeline.