TensorFlow Serving cross-compile project targeting linux on common arm cores from a linux amd64 / x86_64 build host.
The basis of this project is to provide an alternative build strategy for tensorflow/serving with the intention of making it relatively easy to cross-build CPU optimized model server docker images targeting common linux arm platforms. Additonally, a set of docker image build targets is maintained and built for some of the popular linux arm platforms and hosted on Docker Hub.
Upstream Project: tensorflow/serving
Hosted on Docker Hub: emacski/tensorflow-serving
Usage Documentation: TensorFlow Serving with Docker
Note: The project images are desinged to be functionally equivalent to their upstream counter part.
On many consumer / developer 64-bit and 32-bit arm platforms you can simply:
docker pull emacski/tensorflow-serving:latest
# or
docker pull emacski/tensorflow-serving:2.6.0Refer to TensorFlow Serving with Docker for configuration and setting up a model for serving.
| Tag | ARM Core Compatability |
|---|---|
[Version]-linux_amd64_avx_sse4.2 |
N/A |
[Version]-linux_arm64_armv8-a |
Cortex-A35 / A53 / A57 / A72 / A73 |
[Version]-linux_arm64_armv8.2-a |
Cortex-A55 / A75 / A76 |
[Version]-linux_arm_armv7-a_neon_vfpv3 |
Cortex-A8 |
[Version]-linux_arm_armv7-a_neon_vfpv4 |
Cortex-A7 / A12 / A15 / A17 |
Example
# on beaglebone black
docker pull emacski/tensorflow-serving:2.6.0-linux_arm_armv7-a_neon_vfpv3| Alias | Tag | Notes |
|---|---|---|
[Version]-linux_amd64 |
[Version]-linux_amd64_avx_sse4.2 |
default linux_amd64 image |
[Version]-linux_arm64 |
[Version]-linux_arm64_armv8-a |
Should work on most 64-bit raspberry pi and compatible platforms |
[Version]-linux_arm |
[Version]-linux_arm_armv7-a_neon_vfpv4 |
Should work on most 32-bit raspberry pi and compatible platforms |
latest-linux_amd64 |
[Latest-Version]-linux_amd64 |
|
latest-linux_arm64 |
[Latest-Version]-linux_arm64 |
|
latest-linux_arm |
[Latest-Version]-linux_arm |
Examples
# on Raspberry PI 3 B+
docker pull emacski/tensorflow-serving:2.6.0-linux_arm64
# or
docker pull emacski/tensorflow-serving:latest-linux_arm64| Image | OS | Arch |
|---|---|---|
emacski/tensorflow-serving:latest-linux_arm |
linux |
arm |
emacski/tensorflow-serving:latest-linux_arm64 |
linux |
arm64 |
emacski/tensorflow-serving:latest-linux_amd64 |
linux |
amd64 |
Examples
# on Raspberry PI 3 B+
docker pull emacski/tensorflow-serving
# or
docker pull emacski/tensorflow-serving:latest
# the actual image used is emacski/tensorflow-serving:latest-linux_arm64
# itself actually being emacski/tensorflow-serving:[Latest-Version]-linux_arm64_armv8-a| Image | OS | Arch |
|---|---|---|
emacski/tensorflow-serving:[Version]-linux_arm |
linux |
arm |
emacski/tensorflow-serving:[Version]-linux_arm64 |
linux |
arm64 |
emacski/tensorflow-serving:[Version]-linux_amd |
linux |
amd64 |
Example
# on Raspberry PI 3 B+
docker pull emacski/tensorflow-serving:2.6.0
# the actual image used is emacski/tensorflow-serving:2.6.0-linux_arm64
# itself actually being emacski/tensorflow-serving:2.6.0-linux_arm64_armv8-aAs of version 2.0.0, debug images are also built and published to docker hub.
These images are identical to the non-debug images with the addition of busybox
utils. The utils are located at /busybox/bin which is also included in the
image's system PATH.
For any image above, add debug after the [Version] and before the platform
suffix (if one is required) in the image tag.
# multi-arch
docker pull emacski/tensorflow-serving:2.6.0-debug
# specific image
docker pull emacski/tensorflow-serving:2.6.0-debug-linux_arm64_armv8-a
# specific alias
docker pull emacski/tensorflow-serving:latest-debug-linux_arm64Example Usage
# start a new container with an interactive ash (busybox) shell
docker run -ti --entrypoint /busybox/bin/sh emacski/tensorflow-serving:latest-debug-linux_arm64
# with an interactive dash (system) shell
docker run -ti --entrypoint sh emacski/tensorflow-serving:latest-debug-linux_arm64
# start an interactive ash shell in a running debug container
docker exec -ti my_running_container /busybox/bin/shBuild Host Platform: linux_amd64 (x86_64)
Build Host Requirements:
- git
- docker
For each version / release, a self contained build environment devel image is
created and published. This image contains all necessary tools and dependencies
required for building project artifacts.
git clone git@github.com:emacski/tensorflow-serving-arm.git
cd tensorflow-serving-arm
# pull devel
docker pull emacski/tensorflow-serving:latest-devel
# or build devel
docker build -t emacski/tensorflow-serving:latest-devel -f tensorflow_model_server/tools/docker/Dockerfile .All of the build examples assume that the commands are executed within the devel
container:
# interactive shell
docker run --rm -ti \
-w /code -v $PWD:/code \
-v /var/run/docker.sock:/var/run/docker.sock \
emacski/tensorflow-serving:latest-devel /bin/bash
# or
# non-interactive
docker run --rm \
-w /code -v $PWD:/code \
-v /var/run/docker.sock:/var/run/docker.sock \
emacski/tensorflow-serving:latest-devel [example_command]The following bazel config groups represent the build options used for each target
platform (found in .bazelrc). These config groups should be treated as mutually
exclusive with each other and only one should be specified in a build command as
a --config option.
| Name | Type | Info |
|---|---|---|
linux_amd64 |
Base | can be used for custom builds |
linux_arm64 |
Base | can be used for custom builds |
linux_arm |
Base | can be used for custom builds |
linux_amd64_avx_sse4.2 |
Project | inherits from linux_amd64 |
linux_arm64_armv8-a |
Project | inherits from linux_arm64 |
linux_arm64_armv8.2-a |
Project | inherits from linux_arm64 |
linux_arm_armv7-a_neon_vfpv3 |
Project | inherits from linux_arm |
linux_arm_armv7-a_neon_vfpv4 |
Project | inherits from linux_arm |
Build a project maintained model server docker image targeting one of the platforms specified by a project config group as listed above. The resulting image can be found as a tar file in bazel's output directory.
bazel build //tensorflow_model_server:project_image.tar --config=linux_arm64_armv8-a
# or
bazel build //tensorflow_model_server:project_image.tar --config=linux_arm_armv7-a_neon_vfpv4
# each build creates a docker loadable image tar in bazel's output dirSame as above, but additionally bazel attempts to load the resulting image onto the host, making it immediately available to the host's docker.
Note: host docker must be available to the build container for final images to be available on the host automatically.
bazel run //tensorflow_model_server:project_image --config=linux_arm64_armv8-a
# or
bazel run //tensorflow_model_server:project_image --config=linux_arm_armv7-a_neon_vfpv4Build the model server binary targeting one of the platforms specified by a project config group as listed above.
Note: It's not recommended to use these binaries as standalone executables as they are built specifically to run in their respective containers, but they may work on debian 10 like systems.
bazel build //tensorflow_model_server --config=linux_arm64_armv8-a
# or
bazel build //tensorflow_model_server --config=linux_arm_armv7-a_neon_vfpv4Can be used to fine-tune builds for specific platforms. Use a "Base" type
config group and custom compile options. For linux_arm64 and
linux_arm options see: https://releases.llvm.org/10.0.0/tools/clang/docs/CrossCompilation.html
# building an image tuned for Cortex-A72
bazel build //tensorflow_model_server:custom_image.tar --config=linux_arm64 --copt=-mcpu=cortex-a72
# look for custom_image.tar in bazel's output directoryv1.11.1v1.12.0v1.13.0v1.14.0
Note: a tag exists for both v1.14.0 and 1.14.0 as this was the current
upstream tensorflow/serving version when this project was refactored
The following tensorflow serving versions were built using the legacy project structure and are still available on DockerHub.
emacski/tensorflow-serving:[Version]-arm64v8emacski/tensorflow-serving:[Version]-arm32v7emacksi/tensorflow-serving:[Version]-arm32v7_vfpv3
Versions: 1.11.1, 1.12.0, 1.13.0, 1.14.0
This project uses llvm / clang toolchains for c++ cross-compiling. By
default, the model server is statically linked to llvm's libc++. To dynamically
link against gnu libstdc++, include the build option --config=gnulibcpp.
The base docker images used in this project come from another project I maintain called Discolix (distroless for arm).
- Not an ARM expert
- Not a Bazel expert (but I know a little bit more now)
- Not a TensorFlow expert
- Personal project, so testing is minimal
Should any of those scare you, I recommend NOT using anything here. Additionally, any help to improve things is always appreciated.