/TF2DeepFloorplan

TF2 Deep FloorPlan Recognition using a Multi-task Network with Room-boundary-Guided Attention. Enable tensorboard, quantization, flask, tflite, docker, github actions and google colab.

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

TF2DeepFloorplan License: GPL v3 example workflow Coverage StatusHits

This repo contains a basic procedure to train and deploy the DNN model suggested by the paper 'Deep Floor Plan Recognition using a Multi-task Network with Room-boundary-Guided Attention'. It rewrites the original codes from zlzeng/DeepFloorplan into newer versions of Tensorflow and Python.
Network Architectures from the paper,

Additional feature (pygame)

TF2DeepFloorplan_3dviz

Requirements

Depends on different applications, the following installation methods can

OS Hardware Application Command
Ubuntu CPU Model Development pip install -e .[tfcpu,dev,testing,linting]
Ubuntu GPU Model Development pip install -e .[tfgpu,dev,testing,linting]
MacOS M1 Chip Model Development pip install -e .[tfmacm1,dev,testing,linting]
Ubuntu GPU Model Deployment API pip install -e .[tfgpu,api]
Ubuntu GPU Everything pip install -e .[tfgpu,api,dev,testing,linting,game]
Agnostic ... Docker (to be updated)
Ubuntu GPU Notebook pip install -e .[tfgpu,jupyter]
Ubuntu GPU Game pip install -e .[tfgpu,game]

How to run?

  1. Install packages.
# Option 1
python -m venv venv
source venv/bin/activate
pip install --upgrade pip setuptools wheel
# Option 2 (Preferred)
conda create -n venv python=3.8 cudatoolkit=10.1 cudnn=7.6.5
conda activate venv
# common install
pip install -e .[tfgpu,api,dev,testing,linting]
  1. According to the original repo, please download r3d dataset and transform it to tfrecords r3d.tfrecords. Friendly reminder: there is another dataset r2v used to train their original repo's model, I did not use it here cos of limited access. Please see the link here zlzeng/DeepFloorplan#17.
  2. Run the train.py file to initiate the training, model checkpoint is stored as log/store/G and weight is in model/store,
python -m dfp.train [--batchsize 2][--lr 1e-4][--epochs 1000]
[--logdir 'log/store'][--modeldir 'model/store']
[--save-tensor-interval 10][--save-model-interval 20]
[--tfmodel 'subclass'/'func'][--feature-channels 256 128 64 32]
[--backbone 'vgg16'/'mobilenetv1'/'mobilenetv2'/'resnet50']
[--feature-names block1_pool block2_pool block3_pool block4_pool block5_pool]
  • for example,
python -m dfp.train --batchsize=4 --lr=5e-4 --epochs=100
--logdir=log/store --modeldir=model/store
  1. Run Tensorboard to view the progress of loss and images via,
tensorboard --logdir=log/store
  1. Convert model to tflite via convert2tflite.py.
python -m dfp.convert2tflite [--modeldir model/store]
[--tflitedir model/store/model.tflite]
[--loadmethod 'log'/'none'/'pb']
[--quantize][--tfmodel 'subclass'/'func']
[--feature-channels 256 128 64 32]
[--backbone 'vgg16'/'mobilenetv1'/'mobilenetv2'/'resnet50']
[--feature-names block1_pool block2_pool block3_pool block4_pool block5_pool]
  1. Download and unzip model from google drive,
gdown https://drive.google.com/uc?id=1czUSFvk6Z49H-zRikTc67g2HUUz4imON # log files 112.5mb
unzip log.zip
gdown https://drive.google.com/uc?id=1tuqUPbiZnuubPFHMQqCo1_kFNKq4hU8i # pb files 107.3mb
unzip model.zip
gdown https://drive.google.com/uc?id=1B-Fw-zgufEqiLm00ec2WCMUo5E6RY2eO # tfilte file 37.1mb
unzip tflite.zip
  1. Deploy the model via deploy.py, please be aware that load method parameter should match with weight input.
python -m dfp.deploy [--image 'path/to/image']
[--postprocess][--colorize][--save 'path/to/output_image']
[--loadmethod 'log'/'pb'/'tflite']
[--weight 'log/store/G'/'model/store'/'model/store/model.tflite']
[--tfmodel 'subclass'/'func']
[--feature-channels 256 128 64 32]
[--backbone 'vgg16'/'mobilenetv1'/'mobilenetv2'/'resnet50']
[--feature-names block1_pool block2_pool block3_pool block4_pool block5_pool]
  • for example,
python -m dfp.deploy --image floorplan.jpg --weight log/store/G
--postprocess --colorize --save output.jpg --loadmethod log
  1. Play with pygame.
python -m dfp.game

Docker for API

  1. Build and run docker container. (Please train your weight, google drive does not work currently due to its update.)
docker build -t tf_docker -f Dockerfile .
docker run -d -p 1111:1111 tf_docker:latest
docker run --gpus all -d -p 1111:1111 tf_docker:latest

# special for hot reloading flask
docker run -v ${PWD}/src/dfp/app.py:/src/dfp/app.py -v ${PWD}/src/dfp/deploy.py:/src/dfp/deploy.py -d -p 1111:1111 tf_docker:latest
docker logs `docker ps | grep "tf_docker:latest"  | awk '{ print $1 }'` --follow
  1. Call the api for output.
curl -H "Content-Type: application/json" --request POST  \
  -d '{"uri":"https://cdn.cnn.com/cnnnext/dam/assets/200212132008-04-london-rental-market-intl-exlarge-169.jpg","colorize":1,"postprocess":0}' \
  http://0.0.0.0:1111/uri --output /tmp/tmp.jpg


curl --request POST -F "file=@resources/30939153.jpg" \
  -F "postprocess=0" -F "colorize=0" http://0.0.0.0:1111/upload --output out.jpg
  1. If you run app.py without docker, the second curl for file upload will not work.

Google Colab

  1. Click on and authorize access.
  2. Run the first 2 code cells for installation.
  3. Go to Runtime Tab, click on Restart runtime. This ensures the packages installed are enabled.
  4. Run the rest of the notebook.

How to Contribute?

  1. Git clone this repo.
  2. Install required packages and pre-commit-hooks.
pip install -e .[tfgpu,api,dev,testing,linting]
pre-commit install
pre-commit run
pre-commit run --all-files
# pre-commit uninstall/ pip uninstall pre-commit
  1. Create issues. Maintainer will decide if it requires branch. If so,
git fetch origin
git checkout xx-features
  1. Stage your files, Commit and Push to branch.
  2. After pull and merge requests, the issue is solved and the branch is deleted. You can,
git checkout main
git pull
git remote prune origin
git branch -d xx-features

Results

  • From train.py and tensorboard.
Compare Ground Truth (top)
against Outputs (bottom)
Total Loss
Boundary Loss Room Loss
  • From deploy.py and utils/legend.py.
Input Legend Output
--colorize --postprocess --colorize
--postprocess

Optimization

  • Backbone Comparison in Size
Backbone log pb tflite toml
VGG16 130.5Mb 119Mb 45.3Mb link
MobileNetV1 102.1Mb 86.7Mb 50.2Mb link
MobileNetV2 129.3Mb 94.4Mb 57.9Mb link
ResNet50 214Mb 216Mb 107.2Mb link
  • Feature Selection Comparison in Size
Backbone Feature Names log pb tflite toml
MobileNetV1 "conv_pw_1_relu",
"conv_pw_3_relu",
"conv_pw_5_relu",
"conv_pw_7_relu",
"conv_pw_13_relu"
102.1Mb 86.7Mb 50.2Mb link
MobileNetV1 "conv_pw_1_relu",
"conv_pw_3_relu",
"conv_pw_5_relu",
"conv_pw_7_relu",
"conv_pw_12_relu"
84.5Mb 82.3Mb 49.2Mb link
  • Feature Channels Comparison in Size
Backbone Channels log pb tflite toml
VGG16 [256,128,64,32] 130.5Mb 119Mb 45.3Mb link
VGG16 [128,64,32,16] 82.4Mb 81.6Mb 27.3Mb
VGG16 [32,32,32,32] 73.2Mb 67.5Mb 18.1Mb link
  • tfmot
    • Pruning (not working)
    • Clustering (not working)
    • Post training Quantization (work the best)
    • Training aware Quantization (not supported by the version)