nabaztag2018/pynab

ASR flaky with Bullseye

Opened this issue · 17 comments

f-laurens commented

ASR seems flaky on Pi Zero 2:

  • most of the time, recognition fails,
  • and eventually nabd gets stuck (consuming all CPU time) and has to be restarted.
pguyot commented

ASR seemed to work here, didn't test RFID.

Which image did you try with the Zero 2:

  • the rapios zero (supposedly compatible with both zero and zero 2)
  • the raspios zero 2 (optimized for armv7l)
  • a dietpi?
f-laurens commented

So far, I tested with a zero2_dietpi image. I will try with a zero2_raspios image.
I didn't check at all the behaviour of zero images on Pi Zero 2.

pguyot commented

I don't think releasing the zero2_raspios image really makes sense as it might add confusion and the zero_raspios image should work with Zero 2.

f-laurens commented

... the zero_raspios image should work with Zero 2.

Are you sure? I do not quite figure out how the zero Pynab install ( with armv6l Python wheels) will work on an armv7l system...
If you are right, I would then vote for making our builds simpler by not building the zero2 images.

pguyot commented

pip will try and fetch armv7l wheels if the (real or emulated) arch is armv7l, but these wheels install compiled files at the same location as armv6l wheels.

However, on the first upgrade, pip will replace code of these wheels, following requirements.txt logic. This means that after an upgrade, the SD will no longer work on a Zero. Likewise, our upgrade script only recompiles drivers for the native kernel. Not sure about the best behavior here.

f-laurens commented

pip will try and fetch armv7l wheels if the (real or emulated) arch is armv7l, but these wheels install compiled files at the same location as armv6l wheels.

Indeed, but pip is only invoked thru Pynab's install.sh (right?), so a zero (armv6l) image will not work (Pynab wise) 'out of the box' on a Zero 2... (unless we force an upgrade on first boot?)

pguyot commented

A Pynab image created with setup on armv6l will (a) install drivers compiled for armv6l for all three kernels (kernel.img, kernel7.img and kernel7l.img) and (b) install armv6l wheels.

It will boot on Zero 2 which will happily execute this code optimized for armv6l.

On update on Zero 2, pip may replace armv6l code with armv7l code and install.sh may recompile drivers for the current kernel only (kernel7l).

When installed again in a Zero, the native code installed by pip will not work and the drivers will still be the old code compiled for armv6l, but upgrade script will erroneously consider it up to date.

So our image should work in a Zero 2 but putting the SD later in a Zero might not work.

Same with Raspberry Pi OS if you install some native code with pip anyway.

f-laurens commented

A Pynab image created with setup on armv6l will (a) install drivers compiled for armv6l for all three kernels (kernel.img, kernel7.img and kernel7l.img) and (b) install armv6l wheels.

Indeed. I am aware of this.

It will boot on Zero 2 which will happily execute this code optimized for armv6l.

Right, I was blind here, forgetting that the Zero 2 could run armv6l code.... I stand corrected.
In that case, I still fully second your proposal to release (and build) only zero images. It will keep things simpler....

So our image should work in a Zero 2 but putting the SD later in a Zero might not work.

... given that we should not care about this borderline use case.

multiplie-fr commented

ASR is working perfectly here. It's faster too.

ASR seemed to work here, didn't test RFID.

Which image did you try with the Zero 2:

* the rapios zero (supposedly compatible with both zero and zero 2)

* the raspios zero 2 (optimized for armv7l)

* a dietpi?
f-laurens commented

ASR is working perfectly here. It's faster too.

@multiplie-fr: here being what? (what image on what hardware?)

f-laurens commented

I switched from the zero2_dietpi to the zero2_raspios image.
Behaviour is the same:

  • most of the time, recognition fails (seemed better initially than on DietPi, but not really);
  • and in any case eventually nabd gets stuck and has to be restarted.
f-laurens commented

I switched from a zero2 image to a zero image on the Pi Zero 2.
And... behaviour is still the same:

  • most of the time, recognition is flaky (seemed better initially than with zero2 image - experimenter's wishful thinking? - but in fact not);
  • and quickly, after a handful of attempts, nabd gets stuck and has to be restarted.

Also (to be exhaustive) re-running the Pynab install.sh script on the zero image (to replace the originally installed armv6l versions of ASR-related Python packages by armv7l versions) does (as expected) not change the behaviour.

f-laurens commented

Finally, despite all my attempts, it couldn't get ASR to fail with a zero image running on Pi Zero hardware:
it works fine (or at least as well as it did with release v0.9.1); in any case nabd never gets stuck as it eventually does on Pi Zero 2.

f-laurens commented

Concerning this:

However, on the first upgrade, pip will replace code of these wheels, following requirements.txt logic. This means that after an upgrade, the SD will no longer work on a Zero. Likewise, our upgrade script only recompiles drivers for the native kernel. Not sure about the best behavior here.

I have, for the sake of it, played the scenario:

  1. Use SD card with zero image on Pi Zero 2 (bitch about ASR not working...).
  2. Run upgrade.sh script to replace armv6l wheels by armv7l ones (bitch again...).
  3. Replace PI Zero 2 by Pi Zero, using same SD card.
  4. Run upgrade.sh script again to replace armv7l wheels by armv6l ones (enjoy working ASR...).

To make this a little more foolproof in case there was an upgrade to drivers between 1) and 3), we might have the install.sh script build the drivers for all kernels present on the image, as is done in setup.
Then running upgrade.sh would be the only requirement in all cases, when swapping a zero SD card between Pi Zero and Pi Zero 2.

pguyot commented

No significant difference was found for ASR here between a Pi Zero with 0.9.1 and a Pi Zero 2 with latest (Pi Zero compatible) image + two fixes on cr14 and tagtagtag-ears drivers. Taking #36 into account.

f-laurens commented

Thanks for the ears driver fix.
At first it seemed to alleviate a bit the ASR problem on Pi Zero 2 here (I'm currently using a zero image): apparently (?) more successful ASR attempts (taking indeed #36 into account) before failure.
But in the end the problem remains: after a few times, nabd gets stuck during the recognition phase (no 'end of ASR' sound after button is released, and nose LED remains lit), htop shows it consuming all CPU resources, and it has to be restarted.

f-laurens commented

When testing on Pi Zero 2 with the released v1.0.0 image, ASR seems to work fine (nabd does not get stuck).
Since this is a zero_raspbian (Buster / Python 3.7) image, whereas my previous tests were done with zero_raspios or zero_dietpi (Bullseye / Python 3.9) images, it looks like the ASR flakyness on Pi Zero 2 is somehow linked to the Bullseye and/or Python 3.9 environment (or to the build of the ASR-related packages in this environment)....