Feature request: Single file

[Tangerino]

Using a single file for the solution would help existing projects to adopt the solution, mainly due the OTA operations

[BrianPugh]

Hi Tangerino! Can you elaborate a little more on your use case and how a single file would help? Currently it's in multiple files so that (depending on use case), you might not need the entire library on-device.

If we think it significantly helps to have a single file, we could write up some CI to form an amalgamation file on releases.

[Tangerino]

My case is:
I use OTA to upgrade the device.
My files are in a app folder where I install all .mpy files. Keep it simple!
4G connection are not free
Less files = less data traffic, less RTT, smaller error surface. Devices are deployed in Brazil and sometimes in weak signal reception.
Less files is easier to manage than more files, not to mention several subfolders.
That's it

[BrianPugh]

So of your reasons, I think reducing RTT and increasing download simplicity are legitimate. What does your entire pipeline look like? Do you pre-compile the python files to mpy on a server prior to pushing/pulling from a device?

There are just some maintenance obstacles on my side to providing a single-file download.

[Tangerino]

In time.
My whole project works in MacOs and MP without any code change.
This is important due few reasons:

• Development and test in a rich platform
• Code maintenance
• Hardware agnostic (RPi, ESP32 or linux server)

And I just realize that the PC version does not work in MP.

[BrianPugh]

If you include all the files (e.g. vanilla compressor.py and compressor_viper.py), it should work on both CPython and micropython. The reasons for the different implementations:

1. The reference compressor.py is intended to be as simple, straightforward, and pythonic as possible. To convey the idea of the compression algorithm and to serve as a test bed for other implementations.
2. The micropython optimized compressor_viper.py uses micropython-only features (i.e. viper-emitter), isn't very pythonic, is hard to read, but is pretty fast.
3. I'll be coming out with a C implementation soon, as well.

[Tangerino]

hum.

This is not allowed in MP

self._input_buffer = deque(maxlen=self.max_pattern_size)

I did all my tests using the compressor version (no viper in CPython) and all went well until I move it to the ESP32

[Tangerino]

I'll definitely adopt the library. Simple, already fast enough for my performance requirements and good compression rate.
Cheers

[BrianPugh]

I'll see if I can tweak the reference implementations to work on micropython. the deque issue has actually been solved in my c branch (primarily just by including the more fully-featured deque from micropython/micropython-lib#440). Give me a few days to wrap up the c implementation (the code changes have becomes a bit sprawling) and then I'll get back to this.

Glad you'll be adopting the library! Please let me know of any other issues/features you'd like as you use it.

[BrianPugh]

btw, not sure if you've given it a try, but consider trying to use belay for managing your dependencies/micropython-venv, it sounds like your project may benefit from it.

[Tangerino]

I tried to compile the viper version and got this error.
As I never use these advance features, I have no idea what is wrong. If you could give me some hint...

$ mpy-cross -march=xtensawin ./utamp.py 
Assertion failed: (vtype_val == VTYPE_PYOBJ), function emit_native_store_attr, file ../py/emitnative.c, line 1713.
Abort trap: 6

[BrianPugh]

can you explain your environment more? The following commands (mostly) worked for me with a freshly compiled mpy-cross from micropython tag v1.20.0:

$ mpy-cross -march=xtensawin ../../tamp/tamp/__init__.py
$ mpy-cross -march=xtensawin ../../tamp/tamp/compressor_viper.py
$ mpy-cross -march=xtensawin ../../tamp/tamp/decompressor_viper.py
ERROR: xtensa bccz out of range

I suspect the error from compiling decompressor_viper comes from the program being slightly too large for viper. I've only previously tested with the pi pico, which is barely just fit. I will optimize the implementation soon to fit for the xtensawin arch.

[dpgeorge]

There are quite a few limitations with the viper emitter. If you can narrow down the problem then feel free to submit a bug report over at https://github.com/micropython/micropython and we'll try to fix it.

[BrianPugh]

so I investigated it a tiny bit, and basically there's too big of a jump in the code for the xtensawin target. I'll try and rearrange some code so that the jump is smaller (without impacting performance too much)

[BrianPugh]

@Tangerino I have released a new version (v1.0.3) that has a reduced viper decompressor implementation size that now successfully compiles for xtensawin targets. I haven't explicitly tested on an ESP32 device, but it should all work.

If this all works for you, I'll then close this issue.

[BrianPugh]

@dpgeorge for completeness, I don't believe this is any issue with Viper. It's just that xtensawin can only handle an 11-bit jumps (the code says 12 bits, but I'm pretty sure it's really only 11 bits), while the rp2 (my primary test device) can handle 23-bit jumps (I think)? Something along those lines.

[dpgeorge]

I don't believe this is any issue with Viper. It's just that xtensawin can only handle an 11-bit jumps (the code says 12 bits, but I'm pretty sure it's really only 11 bits),

Yes that makes sense, that xtensawin has a limited jump range for conditional jumps (12-bit signed, so 11-bits in each direction).

In principle it should be possible to have a longer jump, eg by doing a short, negative conditional jump over a longer jump. But if you can work around it with your code, that would be preferred.

[BrianPugh]

yup! no issues here, was able to work around it (and make my code a little more efficient 😎 )! Thanks for the clarification on the 11/12 bits!

[BrianPugh]

@Tangerino to celebrate 1 year of this issue, we now have micropython native modules published to releases. This should meet your single-file desire, but each file is platform/micropython-version specific. Please give them a try!