This is a Job DSL project for FAIR/AML's OSSCI infrastructure.
Looking for information about the actual machines the jobs run on? See fairinternal/ossci-infra.
- Get the public access key and secret access key at https://fb.quip.com/oAX3ApaV35jU (Facebook employees only). If you're a non-Facebook employee, talk to @ezyang about getting access.
- Use them as username/password for
aws ecr get-login
(our region isus-east-1
). The aws command comes with awscli; see https://aws.amazon.com/cli/ for more guidance. Once you've done this step once, you no longer need to do it again for that computer. - Search for
docker pull
in your build log and search for something like308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-trusty-py3.6-gcc5.4:tmp-173-5910
; this is your docker image. (If the tag istmp-###-####
, it comes with a build of your source; if it's###
that's the stock image.) - Run
aws ecr get-login
with your AWS credentials to get your Docker login command. Run this command to login.- Use
pip install awscli
to install AWS CLI. - You may need to configure the region first using
aws configure
. Our region isus-east-1
. - If you get the error
unknown shorthand flag: 'e' in -e
, try deleting-e none
from the command line.
- Use
- Run
docker run -it $DOCKER_IMAGE /bin/bash
Try prepending sudo if you get the permission denied
error for the docker commands
(and later figure out why your user doesn't have permissions to connect
to the Docker socket; maybe you need to add yourself to the docker
group and reboot).
Want to run a Docker image on a GPU? Standard issue devgpus don't allow use of Docker, so you will have to either (1) run docker on devfair, (2) get a GPU-enabled AWS instance (the OSS CI team has a few allocated, get in touch with them to see how to connect), (3) find a GPU machine that you're managing yourself. All of these will require some time to provision, so don't try to do this last minute.
Want to know more about what Docker images are available? See "Available docker images."
If you just want to reproduce a test error, there is the particular Docker image for your job which you should pull and test. But if you're interested in repurposing our CI Docker images for other purposes, it helps to know about the general structure of the Docker images our CI exposes and how they are built (so you can find the URL for a base image you might be interested in.)
For historical reasons, there are two sets of Docker images, one for PyTorch and one for Caffe2 (we intend to merge these at some point, but we haven't finished yet.
PyTorch Dockerfiles source lives at https://github.com/pietern/pytorch-dockerfiles and are built every week at https://ci.pytorch.org/jenkins/job/pytorch-docker-master/
Caffe2 Dockerfiles source lives at https://github.com/pytorch/pytorch/tree/master/docker/caffe2/jenkins and are built upon request at https://ci.pytorch.org/jenkins/job/caffe2-docker-trigger/
Summary for gdb-enabled CPU:
ssh ubuntu@$CPU_HOST
docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i $DOCKER_IMAGE /bin/bash
**Summary for ASAN builds (jobs like pytorch_linux_xenial_py3_clang5_asan_test)
ssh ubuntu@$CPU_HOST
docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i $DOCKER_IMAGE /bin/bash
export LD_PRELOAD=/usr/lib/llvm-5.0/lib/clang/5.0.0/lib/linux/libclang_rt.asan-x86_64.so
cd ~/workspace
# run your test repro
Summary for gdb-enabled NVIDIA/CUDA GPU
ssh ubuntu@$GPU_HOST
docker run --rm --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -t -u jenkins -i --runtime=nvidia -e CUDA_VERSION=8 -e NVIDIA_VISIBLE_DEVICES=all $DOCKER_IMAGE /bin/bash
Summary for AMD/ROCM GPU
ssh -p $AMD_PORT $AMD_USERNAME@$AMD_HOST
docker run --device=/dev/kfd --device=/dev/dri --group-add video -it $DOCKER_IMAGE /bin/bash
Get credentials for an AMD GPU machine at https://fb.quip.com/Luj5AQjlH11U This should only be necessary if you actually plan to run tests on an AMD GPU; if you are debugging build failures, any old host is OK, though make sure you have 16G of RAM (at least).
What is my CPU/GPU HOST?
- If you don't need the exact same hardware, you can run these commands on any machine that has Docker
- There are some AWS dev machines which can be used. This particular author likes to use ec2-52-90-201-109.compute-1.amazonaws.com The canonical information about all our running instances can be found on AWS console; you'll need a login under the 'caffe2' account, ask @pietern for access.
What is my Docker image?
Your Docker image will look something like
registry.pytorch.org/pytorch/pytorch-linux-xenial-cuda9-cudnn7-py2:69-3002
.
This image is:
COMMIT_DOCKER_IMAGE
in the build log, andDOCKER_IMAGE
in the test log
You can tell you've got the right one because Jenkins homedir
will have a workspace
directory.
What do I do once I'm in?
Read the actual jobs
directory to see how to actually build/test (at the very least, you will
need to set PATH
to pick up the correct Python executable.)
You DO NOT need to build PyTorch; it will already be installed. But
if you want to inject debugging code - feel free to and use regular python setup.py develop
instructions in ~/workspace
.
What do all the flags in the docker run command mean?
-
The
--rm
argument ensures that the Docker image gets immediately deleted when you exit. If you don't want this, delete--rm
... but don't forget todocker rm
the image when you are done (stopping it is not sufficient.) -
By default Docker does not enable ptrace, which means that you will have a hard time running gdb inside the docker image.
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined
ensures this capability is allowed. -
The docker image will have a pretty bare set of installed packages. To install more, run
sudo apt update
and thensudo apt install
for the packages you want.gdb
andvim
can be quite useful.
The CUDA docker command didn't work.
You need to install nvidia-docker 2.0 which knows how to expose CUDA devices inside Docker.
Where is my source?
Caffe2 builds don't currently store their source code in test images; you will need to git clone a copy of the source and checkout the correct one.
OS X builds are not containerized. You probably have a Macbook; first try reproducing locally. Otherwise, see https://fb.quip.com/FIDAOAi7r2A for canonical information about our OS X workers. (Facebook employees only).
Changes you make to these machines affect everyone, so please be careful.
You must Remote Desktop into the Windows machines; see this Quip for information how to access (you may have to ask for access.)
Changes you make to these machines affect everyone, so please be careful.
Taking PyTorch as an example (much of the same applies to Caffe2), here is how we structure our jobs:
-
The pytorch-docker builds are responsible for taking the Dockerfiles from pytorch-dockerfiles and building Docker images. These images are uploaded to
registry.pytorch.org/pytorch
. Every new built Docker image gets a new tag, which is a sequentially incrementing number.- After a Docker build completes, we test and make sure that master
of PyTorch builds with the new image (in case changes in the image
introduced a regression.) If it passes, the Docker build process
will deploy the image, by making a commit "Update PyTorch DockerVersion"
which updates the latest PyTorch DockerVersion (the tag, really)
in
./src/main/groovy/ossci/pytorch/DockerVersion.groovy
. (You can also manually update the active DockerVersion by editing this file.)
- After a Docker build completes, we test and make sure that master
of PyTorch builds with the new image (in case changes in the image
introduced a regression.) If it passes, the Docker build process
will deploy the image, by making a commit "Update PyTorch DockerVersion"
which updates the latest PyTorch DockerVersion (the tag, really)
in
-
PyTorch builds are intermediated by a top level trigger build for commits to master, and pull requests (using Jenkins GitHub Pull Request Builder).
-
This kicks off parallel builds for each system configuration we are interested in (at the moment, only Python 2 and Python 3, but we will be adding CUDA 8 and CUDA 9 permutations as well.)
-
The configuration build itself is split into two phases. The first phase only builds and installs PyTorch into the Docker image. We then push the Docker image to the registry. This build is done on a CPU-only machine. The second phase tests PyTorch on a GPU provisioned machine by loading the Docker image.
-
-
There are also miscellaneous cronjobs for cleaning old Docker images from the registry and the local builders.
.
├── jobs # DSL script files
├── resources # resources for DSL scripts
├── src
│ ├── main
│ │ ├── groovy # support classes
│ │ └── resources
│ │ └── idea.gdsl # IDE support for IDEA
│ └── test
│ └── groovy # specs
└── build.gradle # build file
./gradlew test
runs the specs.
JobScriptsSpec
will loop through all DSL files and make sure they don't throw any exceptions when processed. All XML output files are written to build/debug-xml
.
This can be useful if you want to inspect the generated XML before check-in.
You can create the example seed job via the Rest API Runner (see below) using the pattern jobs/seed.groovy
.
Or manually create a job with the same structure:
- Invoke Gradle script
- Use Gradle Wrapper:
true
- Tasks:
clean test
- Use Gradle Wrapper:
- Process Job DSLs
- DSL Scripts:
jobs/**/*Jobs.groovy
- Additional classpath:
src/main/groovy
- DSL Scripts:
- Publish JUnit test result report
- Test report XMLs:
build/test-results/**/*.xml
- Test report XMLs:
Note that starting with Job DSL 1.60 the "Additional classpath" setting is not available when Job DSL script security is enabled.
Note: the REST API Runner does not work with Automatically Generated DSL.
A gradle task is configured that can be used to create/update jobs via the Jenkins REST API, if desired. Normally a seed job is used to keep jobs in sync with the DSL, but this runner might be useful if you'd rather process the DSL outside of the Jenkins environment or if you want to create the seed job from a DSL script.
./gradlew rest -Dpattern=<pattern> -DbaseUrl=<baseUrl> [-Dusername=<username>] [-Dpassword=<password>]
pattern
- ant-style path pattern of files to includebaseUrl
- base URL of Jenkins serverusername
- Jenkins username, if securedpassword
- Jenkins password or token, if secured
Sometimes, you will be looking for a function in the Jenkins Job DSL, and it will simply not exist. DO NOT DESPAIR. Read this instead: http://www.devexp.eu/2014/10/26/use-unsupported-jenkins-plugins-with-jenkins-dsl/
In particular, http://job-dsl.herokuapp.com/ is really helpful, even if you're not necessarily working on a custom DSL function.
When you do this, you might want to edit the web UI, and then see the Jenkins? Click on "REST API" at the bottom of the job page and click the link for "config.xml", which will give you the config.xml of the job. Example: https://ci.pytorch.org/jenkins/job/skeleton-pull-request/config.xml
You can navigate to https://ci.pytorch.org/jenkins/script and run Groovy scripts to run ad hoc management tasks. This can be very useful for tasks that are tedious to execute manually.
Beware: with great power comes great responsibility!!
import jenkins.model.*
def folder = Jenkins.instance.items.find { job ->
job.name == "caffe2-builds"
}
def jobs = folder.items.findAll { job ->
job.name =~ /^caffe2-linux-/
}
jobs.each { job ->
println("Planning to remove ${job.name}")
//job.delete()
}
null
def map = [:]
Jenkins.instance.queue.items.each {
i = map.get(it.assignedLabel, 0);
map[it.assignedLabel] = i + 1;
}
sorted = map.sort { a, b -> b.value <=> a.value }
sorted.each { label, count ->
println("${label}: ${count}");
}
println "---"
Jenkins.instance.slaves.each {
println "${it.name} (${it.getComputer().countBusy()}/${it.getNumExecutors()}): ${it.getLabelString()}"
}
null
import hudson.model.*
def queue = Hudson.instance.queue
def cancel = queue.items.findAll {
if (it.task.name.startsWith('name-of-job-to-cleanup')) {
return true;
}
return false;
}
cancel.each {
queue.cancel(it.task)
}
A more pleasant Java development experience can be attained by working on ossci-job-dsl inside a real Java IDE. Here's how to set it up using IntelliJ:
- In the opening splash screen, select "Import Project"
- Select the directory of ossci-job-dsl
- Import project from external model: Gradle
- Click through the last screen, finishing the import
- To test, click "Run" and "Edit configurations"
- Create a new run configuration based on Gradle
- Select the current project as the Gradle project, and put "test" in Tasks.
You now have running tests!