/fruit-finder

An AI fruit scanner for supermarket checkouts

Primary LanguagePython

FruitFinder v3

An AI fruit scanner for supermarket checkouts

Project Overview

Desired Functionality

  • Scan fruit
  • Beep sound when recognised
  • Show picture and name of predicted fruit (and maybe similar altermalives too) for user to confirm
  • User confirms
  • Weigh the fruit and add to cart
  • Repeat

Development Roadmap

  • Collect and annotate fruit images dataset
  • [DONE] Label with label.py & store imgs & labels together on file
  • [DONE] Load training dataset from file w/ dataset.py load_data()
  • [DONE] Load test dataset from file w/ dataset.py load_data()
  • [DONE] Finetute a fast & light object detection algorithm pretrained on ImageNet (Fast RCNN, YOLO, etc.) and save to file w/ train. py
  • [DONE] Load model & run live inference on webcam w/ predict.py
  • Build GUI user interface with sounds and mount raspberry pi for a realistic scanning demo
  • After validating performance on laptop webcam, compress and convert to tflite to be run on a raspberry pi + webcam

TODO:

  • Adapt the tfjs pacman demo to work for my scannner

  • Port my model to tfjs

  • Implement model in tfjs + build a basic UI around it

  • Train multiple fruit and veg classes

  • Implement multiple detections per image plotting

  • Be done with it and move onto other projects

Usage Guide

Installation

See included installation guide for relevant platform.

Training and Running Inference

  • Activate the conda environment by running conda activate fruitfinder
  • Run python train.py to train the model
  • Run python predict.py to test out the trained model

Finding Pretrained Models

Data Labelling (Label Studio <- new way)

Check out their website

Install with:

# Install the package into python virtual environment
pip install -U label-studio
# Launch it!
label-studio

Data Labelling (LabelImg <- old way)

First clone the labelImg repo into ./lib/

git clone https://github.com/heartexlabs/labelImg.git

Installing LabelImg (for MacOS + Anaconda)

conda install pyqt=5
conda install lxml
make qt5py3

From inside labelImg folder:

python -m pip install .

Installing LabelImg (for Windows + Anaconda)

conda install pyqt=5;
conda install -c anaconda lxml;
pyrcc5 -o libs/resources.py resources.qrc;

Then, from inside labelImg folder:

python -m pip install .

See guide on labelImg GitHub for more installation options

Using LabelImg

python labelImg.py
python labelImg.py [IMAGE_PATH] [PRE-DEFINED CLASS FILE]

Converting to TFJS

tensorflowjs_converter --input_format=tf_saved_model --output_node_names='MobilenetV1/Predictions/Reshape_1' --saved_model_tags=serve exported-models\\my_model\\saved_model exported-models\\my_model\\web_model

Other Resources

Model Exporting Methods

Method 1: Use the command line export script Use python .\exporter_main_v2.py --input_type image_tensor --pipeline_config_path .\models\ssd_mobilenet_v2_fpnlite_640x640_totoro\pipeline.config --trained_checkpoint_dir .\models\ssd_mobilenet_v2_fpnlite_640x640_totoro\checkpoint --output_directory .\exported-models\my_model

Method 2: Custom export just the frozen graph .pb

Method 3: Hack engineer a solution using exporter.py

Made with ❤️