Host side code https://github.com/SensorsINI/joker-network and firmware for the Trixsy card finding magic robot.
Trixsy thrusts out its finger at the exactly correct moment as the cards are riffled past its finger.
Trixsy uses a 346x260 DVS event camera designed by the Sensors group (and sold by inivation.com) to generate constant count frames at hundreds of Hz and with a latency of less than 1 ms. Trixy is written in python. Trisy uses https://github.com/duguyue100/pyaer to capture events in a producer process, and sends the pickeled frames to the consumer process by local UDP socket. A TensorflowLite 2.0 CNN detects the selected card in about 4 ms. The PC sends the command to the Arduino Nano, which turns off an NPN bipolar which lets the power MOSFET gate go up to 24V. The MOSFET lets the charge stored on an 11F ultra cap array charged up to 48V dump onto a solenoid to push our the finger lever in less than 15ms. Trixsy's whole finger can be powered by USB bus power (24V comes from 5V VBUS via $2 DC-DC converter) although for quicker finger we sometimes use a cheap DC power supply that supplies up to 60V (we use two ultracap arrays in series in this case).
Trixsy was developed by Tobi Delbruck, visiting CSC student Shasha Guo, and PhD students Min Liu and Yuhuang Hu. Thanks for filming to Joao Sacramento and Johannes Oswald.
See https://sensors.ini.uzh.ch for latest news and more demos
Test environment:
- OS: Fully tested on Ubuntu 18.04, partially on Mac OSX 10.15.7, can train on Windows 10 but not run pyaer to run demo
- python: 3.8
- CUDA toolkit 10.1 https://developer.nvidia.com/cuda-toolkit-archive (you might need to downgrade from latest v10.2 if dymamic library cannot load, TF2.3.1 is only compatible with CUDA 10.1 as of this writing)
- tensorflow: 2.3.1
- Keras: 2.3.1
- pyaer https://github.com/duguyue100/pyaer
Make a conda environment, activate it, then in it install the libraries.
pip install tqdm opencv-python tensorflow keras pyserial engineering_notation matplotlib sklearn psutil tensorflow-addons
pip install pyaer # only on linux
pip install flopco-keras # only from clone/fork of https://github.com/tobidelbruck/flopco-keras
For some reason, pip is preferred over conda for installing opencv and tensorflow. At least at time of this writing.
pyaer needs https://github.com/inivation/libcaer. Clone it, then follow instructions in its README to install libcaer.
a. Error [Errno 40] Message too long. This is caused by too small UDP packet size is set, default value is only 9216. Use this command to change the value to 65535: sudo sysctl -w net.inet.udp.maxdgram=65535
b. os.environ['KMP_DUPLICATE_LIB_OK']='True' might be required to enable duplicate openMPs running, Otherwise there might be some errors about openMP reported.
- connect hardware: DVS to USB and Arduino to USB.
- Find out which serial port device the Arduino appears on. You can use dmesg on linux. You can put the serial port into globals_and_utils.py to avoid adding as argument.
- In first terminal run producer
python -m producer- In a second terminal, run consumer
python -m consumer arduinoPort
example: python -m consumer.py ArduinoControl/trixsie-firmware/trixsie-firmware.ino
Output on startup on serial port is
*** Trixsie Oct 2020 V1.0
Compile date and time: Oct 18 2020 17*** Trixsie Oct 2020 V1.0
Compile date and time: Oct 18 2020 17:51:49
Compiled with DEBUG=false
Finger pulse time in ms: 150
Finger hold duty cycle of 255: 30
Send char '1' to activate solenoid finger, '0' to relax it
+/- increase/decrease pulse time, ]/[ increase/decrease hold duty cycle
train.py trains the network
python train.py
TRAIN_DATA_FOLDER is where your examples are organized with three sub-folders: train/, valid/, and test/.
The script make_train_valid_test() in dataset_utils.py builds these folders using desired split from source folders class1 and class2.
Each train/valid/test sub-folder contains two sub-folders, class1/ and class2/.
class1/ has non-joker images and class2/ contains joker images.
A snapshot of the dataset is stored on sensors-nas in the folder sensors/trixsy.
2020-10-22 22:34:29.078718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
dataset path: TRAIN_DATA_FOLDER=/home/tobi/Downloads/trixsyDataset
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1 (Conv2D) (None, 54, 54, 64) 7808
_________________________________________________________________
batch_normalization (BatchNo (None, 54, 54, 64) 256
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 26, 26, 64) 0
_________________________________________________________________
conv2 (Conv2D) (None, 26, 26, 64) 102464
_________________________________________________________________
batch_normalization_1 (Batch (None, 26, 26, 64) 256
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 12, 12, 64) 0
_________________________________________________________________
conv3 (Conv2D) (None, 12, 12, 128) 73856
_________________________________________________________________
conv4 (Conv2D) (None, 12, 12, 128) 147584
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 5, 5, 128) 0
_________________________________________________________________
flatten (Flatten) (None, 3200) 0
_________________________________________________________________
fc8 (Dense) (None, 100) 320100
_________________________________________________________________
dropout (Dropout) (None, 100) 0
_________________________________________________________________
output (Dense) (None, 2) 202
=================================================================
Total params: 652,526
Trainable params: 652,270
Non-trainable params: 256
Found 33555 images belonging to 2 classes.
Found 3018 images belonging to 2 classes.
Found 0 images belonging to 2 classes.
1/300 [..............................] - ETA: 0s - loss: 2.1360 - accuracy: 0.2969
263/300 [=========================>....] - 118s 448ms/step - loss: 0.4770 - accuracy: 0.8034 - val_loss: 0.5494 - val_accuracy: 0.8131
2020-10-22 22:36:28.053056 __main__ - INFO - saving model to {filename} (joker-classify-network.py:297)
See training.log.
== Timing statistics ==
overall consumer loop n=20772: 11.14ms +/- 35.28ms (median 6.50ms, min 2.57ms max 2.11s)
recieve UDP n=20772: 4.79ms +/- 35.09ms (median 25.03us, min 14.31us max 2.11s)
unpickle and normalize/reshape n=20771: 141.34us +/- 38.75us (median 131.61us, min 82.02us max 625.37us)
run CNN n=20771: 4.04ms +/- 894.48us (median 3.91ms, min 2.37ms max 10ms)
###producer
== Timing statistics ==
overall producer frame rate n=32463: 10.62ms +/- 66.40ms (median 5.43ms, min 1.21ms max 3.98s)
2020-10-23 12:02:37,130 - __main__ - INFO - closing <pyaer.davis.DAVIS object at 0x7fe47768e370> (producer.py:28)
2020-10-23 12:02:37,136 - globals_and_utils - INFO - saving timing data for overall producer frame rate in numpy file data/producer-frame-rate-20201023-1156.npy (globals_and_utils.py:101)
2020-10-23 12:02:37,136 - globals_and_utils - INFO - there are 32463 times (globals_and_utils.py:102)
accumulate DVS n=32463: 8.46ms +/- 66.02ms (median 3.73ms, min 154.50us max 3.97s)
normalization n=32462: 1.33ms +/- 329.35us (median 1.27ms, min 754.12us max 4.11ms)
send frame n=32462: 104.71us +/- 29.46us (median 100.61us, min 41.01us max 397.44us)
show DVS image n=2543: 4.58ms +/- 1.66ms (median 4.37ms, min 2.45ms max 64.55ms)