This is a simple example of utilizing distributed training with Tensorflow.
tf.distribute.experimental.MultiWorkerMirroredStrategy is used for synchronous training among all workers, and the connection among the workers is established via gRPC with tf.distribute.experimental.CollectiveCommunication.RING. Number of workers is preset to 3 for demonstration simplicity. Dataset sharding is automatically handled by the distribute API.
Since the training and distribution is handled within docker containers for each worker, a bridge network is required. After build.sh generates the docker container, it creates the shared network bridge - sk_tfdistnetwork.
Each container are given unique names (worker0,...) by run.sh, so that it is exposed to other containers. TF_CONFIG environment variables for each container are set correspondingly.
git clone https://github.com/skmono/dist_mnistdemo.git
cd dist_mnistdemo
./build.sh
This builds and docker image and generates the network bridge.
Note: If sudo is required to run docker commands, run the scripts (build.sh and run.sh with sudo privileges.)
To test the distributed training, run the first worker with
./run.sh 0
and other workers in different terminals (2 more required) with
./run.sh 1
./run.sh 2
The designated names will be removed after training is complete.