/job2docker

Scripts for containerizing Talend Jobs

Primary LanguageShellApache License 2.0Apache-2.0

Talend Docker Example

This repository includes sample bash scripts and utilities which allow you to deploy Talend jobs to Docker containers.

The job2docker approach converts a single Talend job zip file to a container. The resulting Docker image will have a single entry point for the job. It is intended for use by developers during their build / test / debug cycle and provides desktop parity.

Job2Docker Workflow

  1. A Talend job2docker_listener job is used to monitor a shared directory.
  2. The developer clicks Build in Talend Studio to create Talend job zip file in the shared directory.
  3. The Talend job2docker_listener triggers the job2docker script to convert the Talend zip file to a tgz ready for Docker.
  4. The Talend job2docker_listener triggers the job2docker_build script.
  5. The Talend job2docker_listener optionally publishes the resulting container to a Docker Registry.

Job2docker can be incorporated into a CI build environment, but it is out of scope for this cookbook. When run as part of the CI build script, job2docker will presumably run on a CI server local to Nexus and the SCM. Running job2docker in such a CI server will create a Docker image and supporting artifacts closer to the Docker registry so there will be less network overhead than transferring from a laptop.

Prerequisites

Setup

A setup script is provided for linux.
The script will clone the job2docker directory and working directories under a target directory which defaults to ${HOME}/talend.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup)

Jobs built by Studio are dropped into a shared directory that defaults to ${HOME}/shared_jobs. This can be overridden by passing a parameter.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup) ${HOME}/my_shared_jobs

The files can be installed under an alternate directory by passing a second parameter to the script. Note that the shared job directory parameter must also be supplied as the first argument.

    bash <(curl https://raw.githubusercontent.com/Talend/job2docker/master/job2docker-setup) ${HOME}/shared_jobs ${HOME}/mytalend

Launch the job2docker_listener from the target directory.

    ${HOME}/talend/job2docker_listener

Manual install instructions are available if you prefer.

Environment

  • All containerization work is done on Linux with Docker installed.

  • Talend Studio steps can run on a separate machine if desired, it could be a Windows machine.

  • A common drop point (shared directory, shared network drive, shared folder) for Jobs built from Studio to be processed by job2docker.

  • The environment used to test these scripts was a Windows laptop running Talend Studio.

  • The docker scripts were run on an Ubuntu 16.04.2 LTS VM running kernel 4.4.0-97-generic.

  • It was also tested with a Centos 7 VM running kernel 3.10.0-862.6.3.el7.x86_64.

  • VirtualBox was used for the VM hosting.

  • A shared folder was created using VirtualBox so that Studio builds would be visible to the Linux VM.

Directory Index

  • job2docker/job2docker-setup - install script
  • job2docker/bin - scripts for creating docker images, creating containers, and deploying images to the cloud
  • job2docker/docs - additional readme files
  • job2docker/job2docker_build - sample Dockerfile used to create Docker image containing the Talend job
  • job2docker/util - utility bash scripts
  • job2docker/jobs - sample jobs
  • j2d - created during setup, working directory for running the agent that monitors the build directory for Talend zip files

Getting Started