A locally running render farm with Deadline.
Table of contents
- Overview
- Motivation
- Prerequsities
- Usage
- Submitting Jobs
- Running Arbitrary Commands
- Interactive Use
- Windows
- Deadline 9
This project proivides a locally running render farm with Deadline and is based on the article "Running Deadline 8 in Docker Part I | Part II" with the added bonus of encapsulating the entire system, including 2 slaves, via Docker Compose.
Tested on Windows and Linux.
Developing for a distributed mechanism is akin to relying on the weather - you can never be sure what's out there or for how long it'll remain. It complicates life for a fellow developer interested in putting together a pipeline involving something like Deadline.
This project is meant as a local development silo for when you develop against Deadline, with the expectation that once you're finished you'll deploy into production and have things work the same, but without the uncertainty or surrounding infrastructure required to get Deadline off the ground.
The goal of this project is to expose a private render farm for purposes of pipeline development. As such, in order to submit renders to 2 slaves running Autodesk Maya 2016, here's what (all) you need.
- Docker
- Official Deadline installer
Testing installation
This process assumes you have access to Docker via your terminal.
$ docker --version
# Docker version 17.05.0-ce, build 89658beUse consists of (1) cloning this repository, (2) installing a Docker volume for the Deadline Repository, (3) building a Deadline client container and (4) putting it all together via Docker Compose.
Docker Compose is responsible for orchestrating the 5 pieces required for Deadline to function correctly and gluing them together accordingly.
- mongodb
- slave1
- slave2
- webservice
- samba
$ git clone https://github.com/mottosso/docker-deadline.gitNow put the official installers into the installers/ directory.
Example
docker-deadline/
installers/
DeadlineClient-8.0.17.2-linux-x64-installer.run
DeadlineRepository-8.0.17.2-linux-x64-installer.runNote that this project requires version 8 of Deadline, I didn't manage to get 9 working. See below.
Now we can go ahead and initialise the repository volume
$ cd docker-deadline
$ ./install.shThis will install the Deadline Repository into a dedicated Docker volume called deadline-volume. Clients are then handed this volume upon startup.
You can check to see whether the volume was created correctly via docker volume.
$ docker volume list
DRIVER VOLUME NAME
local deadline-volumeNow let's kick things off via two machines with Maya 2016 pre-installed.
$ ./upPro Tip Keep in mind that each time you boot up Deadline, it'll produce a new instance of MongoDB which is where jobs are kept. The repository on the other hand is persistent as a Docker Volume. You can inspect the contents of this volume by accessing the default Samba share accessible via the IP address to the host of your container;
localhoston Linux.
At this point, you'd probably want to launch Deadline Monitor.
$ ./deadlinemonitor.shAnd you're done. See Submitting Jobs for how to actually submit a job.
Now that Deadline and all of its components are up and running, you're able to submit jobs to it via the RESTful interface.
import json
import getpass
import requests
url = "http://localhost:8082/api/jobs"
payload = {
"JobInfo": {
"Name": "My render",
"UserName": getpass.getuser(),
"Plugin": "MayaBatch",
"Frames": "1000-1003x1",
},
"PluginInfo": {
"SceneFile": "/share/my_file.ma",
"Version": "2016",
"OutputFilePath": "/share"
},
"AuxFiles": []
}
payload = json.dumps(payload)
requests.post(url, data=payload)Under Docker Toolbox, you'll need to replace
localhostwith the IP to the virtual machine running your Docker Host.
You'll notice I've included an example Maya file. Once submitted, the Docker slaves will produce an image much like the one below.
You can inspect the contents of the Deadline Repository via the provided Samba container.
$ ls \\<your-ip>\\DeadlineRepository8It will be accessible like any other shared network folder.
Pro Tip If you are running Docker Toolbox, the IP is accessible via a call to
docker-machine ip.
The run.sh provides a command-line interface over the internals of the client container. It takes the following syntax.
$ ./run.sh name-of-container name-of-executableFor example
$ ./run.sh slave1 ./deadlinelauncherGain access to the bash-prompt from within the Client container like so.
$ ./run.sh interactive bashThe farm typically runs at less than 1% of CPU capacity on my moderate, 5-year old laptop. You can kill it (gracefully) like this.
$ ./down.shThe above assume a Linux environment. In order to run Deadline Monitor on Windows, you need a X11 server running, such as VcXsrv or Xming.
With VcXsrv running, running deadlinemonitor.sh should work well with default settings. The container is being passed $HOSTNAME from your host to represent the name with which to connect to. If this variable isn't set, you can set it your self, either to your local hostname or IP address.
$ set HOSTNAME=192.168.0.12This project containerises Deadline 8 because of an issue with the client installation under 9.
Error: An error occurred while trying to set the repository connection settings.
You may have to provide them again when running the client.$ ./deadlinemonitor
QXcbConnection: Could not connect to display my_hostname:0
...
Deadline Monitor will now exit.- Option A) Ensure X11 is running on your computer. Under Windows, see VcXsrv or Xming.
- Option B) Trade
$HOSTNAMEfor your IP address inrun.sh, sometimes Docker fails to resolve a hostname to an IP address
Sometimes Deadline Monitor quits when running docker-compose up due to Mongo not having had enough time to initialise.
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.99.100', port=8082): Max retries exceeded with url: /api/jobsOne solution to this is to simply bring it down, and back up again.