scito/extract_otp_secrets

[suggestion] Upgrade to Nuitka instead of PyInstaller

SirDank opened this issue · 8 comments

  • Improved performance: Nuitka can significantly improve the performance of Python code by compiling it to a binary format that can be executed more efficiently than interpreted code. This can result in faster execution times and better resource utilization.
scito commented

Thanks for the suggestion. I did not know Nuitka(GitHub). I'll have a look at it. I'm wondering if Nuitka can also include binary dependencies as PyInstaller.
I think speed of the extract_otp_secrets application is secondary as it is a "one time usage" application.

You can make one-time-use applications with nuitka similar to pyinstaller with the --onefile option and you should be able to include any type of file as well.

scito commented

I've tested it locally:

python3.11 -m nuitka --enable-plugin=tk-inter --enable-plugin=pyqt5 --include-data-dir=$HOME/.local/__yolo_v3_qr_detector/=__yolo_v3_qr_detector/ --onefile src/extract_otp_secrets.py

Nuitka is working, however the performance difference to PyInstaller is not very big for time ./extract_otp_secrets.bin -h on my i7-4500U 1.8GHz (Haswell, 4th Gen) computer:

Type Startup time
PyInstaller real 0m2.142s, user 0m1.942s, sys 0m0.501s
Nuitka real 0m1.789s, user 0m1.378s, sys 0m0.708s

The build times of pyinstaller and nuitka are very different: Nuitka is about 30x slower to build, 36min instead of 1min.

Measured on my i7-4500U 1.8GHz (Haswell, 4th Gen) computer:

Type Build time
PyInstaller real 1m14.388s, user 0m59.825s, sys 0m2.429s
Nuitka real 35m45.049s, user 25m49.055s, sys 0m59.472s

Currently, it is not feasible to use Nuitka for building.

That difference is wild, could it be possible you're missing some flags that speed things up, for example: --noinclude-default-mode=nofollow

If you need me to try and make a nuitka build and see how long it takes, just shoot me a message!

I rerun it (did not clear the caches):

Type Build time
PyInstaller real 0m1.546s, user 0m0.678s, sys 0m0.416s
Nuitka with flag real 8m0.421s,user 14m28.326s, sys 0m14.239s

Cleaned caches, but not ccache:

Type Build time
PyInstaller real real 1m13.567s, user 0m57.239s, sys 0m2.090s
Nuitka with flag real 6m5.396s, user 7m23.616s, sys 0m7.716s

So, it looks much better.

I get some warnings:

Nuitka-Plugins:WARNING: pyqt5: For the obsolete PyQt5 the Nuitka support is incomplete. Threading, callbacks to compiled
Nuitka-Plugins:WARNING: functions, etc. may not be working.
Nuitka-Plugins:WARNING:     Complex topic! More information can be found at https://nuitka.net/info/pyqt5.html
Nuitka: Starting Python compilation with Nuitka '2.3' on Python '3.11' commercial grade 'not installed'.
Nuitka-Plugins:anti-bloat: Not including '_bisect' automatically in order to avoid bloat, but this may cause: may slow down by
Nuitka-Plugins:anti-bloat: using fallback implementation.
Nuitka-Plugins:anti-bloat: Not including '_json' automatically in order to avoid bloat, but this may cause: may slow down by
Nuitka-Plugins:anti-bloat: using fallback implementation.
Nuitka-Plugins:WARNING: pyqt5: Unwanted import of 'tkinter' that is redundant with 'PyQt5' encountered. Use
Nuitka-Plugins:WARNING: '--nofollow-import-to=tkinter' or uninstall it for best compatibility with pure Python execution.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.                                        
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: gcc (gcc 7).
Nuitka-Scons: Backend linking program with 257 files (no progress information available for this stage).
Nuitka-Scons: Compiled 256 C files using ccache.
Nuitka-Scons: Cached C files (using ccache) with result 'cache hit': 241
Nuitka-Scons: Cached C files (using ccache) with result 'cache miss': 15
Nuitka-Plugins:dll-files: Found 1 file DLLs from cv2 installation.

A try with clang:

Type Build time
Nuitka cold real 12m8.076s, user 28m43.976s, sys 0m32.064s
Nuitka warm real 5m24.256s, user 5m0.535s, sys 0m4.787s

Here's some more flags you could use:
--clang --warn-unusual-code --assume-yes-for-downloads --show-anti-bloat-changes --plugin-enable=pylint-warnings --noinclude-pytest-mode=nofollow --noinclude-setuptools-mode=nofollow --noinclude-default-mode=nofollow

Onefile:
--onefile --enable-plugin=upx --upx-binary=path_goes_here --onefile-tempdir-spec={TEMP}\\custom_name_{PID}