Processing Block template

This is a template for building processing blocks.

# File that defines the docker image to be built - most likely you don't have to change this file.
├── Dockerfile
# Make utilities, defines what happens when running `make something`. Change the name of the docker image in the DOCKER_TAG variable.
├── Makefile
├── README.md
# The manifest of this block (see validate manifest section for more). Change name and other meta properties.
├── UP42Manifest.json
# A coverage badge. Shows the test coverage - should be green.
├── coverage.svg
# The end to end test for this block. Ensures docker image is built built correctly. Change image name to set to the DOCKER_TAG.
├── e2e.py
# Defines python linter ruleset.
├── pylintrc
# Python requirements for development - includes black, pytest, pylint, etc.
├── requirements-dev.txt
# Actual block python requirements shipped in docker image.
├── requirements.txt
# Folder that should include all the code for this block.
├── src
        # Main block code. Add more modules if needed.
│   ├── processing_block.py
        # This is the block entry point - should call `run` method.
│   └── run.py
# Bash file that defines test run (runs pytest with settings, formats code, generates coverage badge)
├── test.sh
# Folder that includes all test code.
└── tests
        # Where you can import all needed modules for testing.
    ├── context.py
        # Actual tests. Recommend one test file per module.
    └── test_processing_block.py

Requirements

This template requires the Mac or Ubuntu bash. In order to use this template to create blocks for the UP42 platform the following tools are required:

UP42 account - Sign up for free!
Python 3.7
A virtual environment manager e.g. virtualenvwrapper
git
docker engine
GNU make
Google Cloud SDK for fetching data for end to end tests

Instructions

The following step-by-step instructions will guide you through setting up, dockerizing and pushing a block using this template to UP42.

Installing the required libraries

First create a new virtual environment called up42-processing-block, for example by using virtualenvwrapper:

# Use the block name as the environment name
mkvirtualenv --python=$(which python3.7) up42-processing-block

Activate the new environment:

workon up42-processing-block

Install the necessary Python libraries via:

make install

If you encounter issues with running make commands in MacOS (such as an xcrun: error: invalid active developer path), consider following the instructions in this thread.

Testing the block locally

Before uploading the block to the UP42 platform, we encourage you to run local tests and validations to ensure that the block works as expected, conforms to the UP42 specifications and could be successfully applied in a UP42 workflow.

Run the unit tests

By successfully running the implemented Python unit tests you can ensure that the block processing functionality works as expected. This template uses pytest for testing, which was installed in the previous step. Run the unit tests via:

make test

Validate the manifest

Then test if the block manifest is valid. The UP42manifest.json file contains the block capabilities. They define what kind of data a block accepts and provides, which parameters can be used with the block etc. See the UP42 block capabilities documentation. Validate the manifest via:

make validate

Run the end-to-end test

In order to run the final end-to-end (e2e) test the block code needs to be dockerized (put in a container that later on would be uploaded to UP42). The end-to-end test makes sure the block's output actually conforms to the platform's requirements. To run the e2e tests it is necessary to have gsutil installed which is part of the Google Cloud SDK.

First build the docker image locally.

make build

Run the e2e tests with:

make e2e

Pushing the block to the UP42 platform

First login to the UP42 docker registry. <USERNAME> needs to be replaced by your UP42 username, which is the email address you use on the UP42 website.

docker login -u=<USERNAME> http://registry.up42.com

# Example:
docker login -u=hans.schmidt@up42.com http://registry.up42.com

In order to push the block to the UP42 platform, you need to build the block Docker container with your UP42 USER-ID. To get your USER-ID, go to the UP42 custom-blocks menu. Click on "PUSH a BLOCK to THE PLATFORM" and copy your USERID from the command shown on the last line at "Push the image to the UP42 Docker registry". The USERID will look similar to this: 63uayd50-z2h1-3461-38zq-1739481rjwia

Replace <User-ID> with your UP42 User-ID.

docker build . -t registry.up42.com/<USER-ID>/a-processing-block:1.0 --build-arg manifest="$(cat UP42Manifest.json)"

# Example:
docker build . -t registry.up42.com/6760d08e-54e3-4f1c-b22e-6ba605ec7592/a-processing-block:1.0 --build-arg manifest="$(cat UP42Manifest.json)"

Now you can finally push the image to the UP42 docker registry. Replace <User-ID> with your UP42 User-ID.

docker push registry.up42.com/<USER-ID>/a-processing-block:1.0

# Example:
docker push registry.up42.com/6760d08e-54e3-4f1c-b22e-6ba605ec7592/a-processing-block:1.0

Success! The block will now appear in the UP42 custom blocks menu menu and can be selected under the Custom blocks tab when building a workflow.

Support, questions and suggestions

Open a github issue in this repository; we are happy to answer your questions!

felytic/up42_aspectum_block