After installation of aXeleRate, test_training_inference.py are freezing

Question

After installation of aXeleRate, test_training_inference.py are freezing

orossant opened this issue 4 years ago · 2 comments

Describe the bug
I have followed your very helpfull and clear procedure explained here : https://www.instructables.com/Object-Detection-With-Sipeed-MaiX-BoardsKendryte-K/

After having installed conda, activating an environment and installing aXelerate on my Mac, I have launched some tests to check all is fine. So I have made different test :
python ./tests_training_inference.py
python ./tests_training_inference.py -t classifier -a 'Tini Yolo'
python ./tests_training_inference.py -t classifier -a 'Full Yolo'
there is always the same error : Epoch 1/5 is ok, but when it comes to Epoch 2/5, it is freezing at 1/5 steps

To Reproduce
Steps to reproduce the behavior:
just follow https://www.instructables.com/Object-Detection-With-Sipeed-MaiX-BoardsKendryte-K/

Expected behavior
A clear and concise description of what you expected to happen.

Expected behavior : be able to run Epoch 1 to 5 without any errors or freeze

Screenshots
Epoch 1/5
5/5 [==============================] - 8s 1s/step - loss: 1.6890 - accuracy: 0.2889 - val_loss: 1.6095 - val_accuracy: 0.2000

Epoch 00001: val_accuracy improved from -inf to 0.20000, saving model to projects/classifier/2021-01-08_15-34-56/Classifier_best_val_accuracy.h5
Epoch 00000: Learning rate is 2.6666666666666667e-05.

Epoch 2/5
1/5 [=====>........................] - ETA: 6s - loss: 1.4810 - accuracy: 0.5000

Environment (please complete the following information):
environment local : MacOs X Catalina, miniconda 3 installed and one dedicated environment activated
conda create -n yolo python=3.7
conda activate yolo
pip install git+https://github.com/AIWintermuteAI/aXeleRate
inside aXeleRate folder
python ./tests_training_inference.py

Additional context
Add any other context about the problem here.
I have found many people have similar pbs but in different context
https://www.google.com/search?client=firefox-b-e&q=keras+freeze+during+training

Answer 1 · 2021-01-08T15:22:03.000Z

additional info to give all package version inside my environment

conda list  ✔  yolo2   16:16:45 

packages in environment at /Users/orossant/miniconda3/envs/yolo2:

Answer 2 · 2021-01-10T11:17:54.000Z

Hello!
Well, the main issue that I'm seeing here is that you're using Mac OS. aXeleRate is meant to be run (and tested only) on Linux(Ubuntu 18.04) and Google Colab. While training theoretically should work on both Windows and Mac OS, aXeleRate is primarily meant as framework for training AND conversion of models to be run on embedded devices. The conversions step utilizes various converters , some of them (such as Google Edge TPU model converter) do no run anywhere except Linux, and some others(such as nncase)are unstable and buggy on Win/Mac.
I need to add this to README :)

Meanwhile, you can try running training in Google Colab, where you can utilize GPUs. Alternatively, if you want to run aXeleRate locally on Mac computer, you can install and run it in virtual machine.

packages in environment at /Users/orossant/miniconda3/envs/yolo2:

Name Version Build Channel