Introduction

This project is to show how to detect and recognize buttons in an elevator for robotics.

Button Detection: tensorflow(<2.0) detection API
Button Recognition: OCR
Button Status (On/Off): Mean color value of each button

Environment

    $ conda create -n detection python=3.7 pyqt=5
    $ conda activate detection  
    (detection)$ git clone https://github.com/supertigim/elevator_buttons_recognition.git  
    (detection)$ cd elevator_buttons_recognition
    (detection)elevator_buttons_recognition$ pip install -r requirements.txt  
    (detection)elevator_buttons_recognition$ mkdir addons && cd addons  
    (detection)elevator_buttons_recognition/addons$ git clone https://github.com/tzutalin/labelImg.git  
    (detection)elevator_buttons_recognition/addons$ cd labelImg  
    (detection)elevator_buttons_recognition/addons/labelImg$ pip install -r requirements/requirements-linux-python3.txt
    (detection)elevator_buttons_recognition/addons/labelImg$ cd ../..
    (detection)elevator_buttons_recognition$ git clone https://github.com/tensorflow/models.git

When labelImge doesn't work properly,

    (detection)elevator_buttons_recognition/addons/labelImg$ sudo apt-get install pyqt5-dev-tools  
    (detection)elevator_buttons_recognition/addons/labelImg$ sudo pip3 install -r requirements/requirements-linux-python3.txt  
    (detection)elevator_buttons_recognition/addons/labelImg$ make qt5py3  
    pyrcc5 -o libs/resources.py resources.qrc

Training

Before training, environment needs to be setup.

    (detection)elevator_buttons_recognition$ sudo apt-get install protobuf-compiler
    (detection)elevator_buttons_recognition$ cd models/research
      
    # Once done, Don't need to do again
    (detection)elevator_buttons_recognition/models/research$ wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
    (detection)elevator_buttons_recognition/models/research$ unzip protobuf.zip
    (detection)elevator_buttons_recognition/models/research$ protoc object_detection/protos/*.proto --python_out=.
      
    # For each terminal or put it in .bashrc for convenience
    (detection)elevator_buttons_recognition/models/research$ export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

There are 5 steps with the additional step for monitoring

    # 1. Create xmls with labelImg
    (detection)elevator_buttons_recognition$ python ./addons/labelImg/labelImg.py 

    # 2. Convert xml to csv 
    (detection)elevator_buttons_recognition$  python xml_to_csv.py -i ./images/train/ -o ./annotations/train_labels.csv
    (detection)elevator_buttons_recognition$ python xml_to_csv.py -i ./images/test/ -o ./annotations/test_labels.csv

    # 3. Convert .csv to .record
    (detection)elevator_buttons_recognition$ python generate_tfrecord.py --csv_input=./annotations/train_labes.csv --output_path=./annotations/train.record --img_path=images/train/
    (detection)elevator_buttons_recognition$ python generate_tfrecord.py --csv_input=./annotations/test_labes.csv --output_path=./annotations/test.record --img_path=images/test/

    # 4. Start training   
    (detection)elevator_buttons_recognition$ python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_inception_v2_coco.config

    # (Optional) For visualization 
    (detection)elevator_buttons_recognition$ tensorboard --logdir=training

    # 5. Conversion to .pb file
    (detection)elevator_buttons_recognition$ python freeze_model.py --input_type image_tensor --pipeline_config_path ./training/ssd_inception_v2_coco.config --trained_checkpoint_prefix ./training/model.ckpt-200000 --output_directory ./frozen_model

Inference

    (detection)elevator_buttons_recognition$ python main.py -m cam      # Camera Streaming
    # or 
    (detection)elevator_buttons_recognition$ python main.py -m image    # Images Files
    # or 
    (detection)elevator_buttons_recognition$ python main.py -m video    # Video File

The pressed button on the image is recognized in red.

To-do List

Press Button Detection Improvement
Tensorflow 2.0 Implementation using Model Garden

Reference

How to build my own button detector

Papers in regard to elevator buttons detection

ETC