echo-lalia/MicroHydra

utf8 doesn't work when frozen into MicroPython

Closed this issue · 7 comments

echo-lalia commented

Kanji support has recently been added thanks to contributions from RealClearwave.

However, the Kanji module relies on a .txt file in the /font directory. The module reads the data as needed from the flash, and doesn't store the entire file in RAM (which makes sense because the file is fairly large).
However, .txt files can't be simply frozen into the MH firmware (as a .bin file) the same way .py files can be. So, in order for this module to work when frozen, it will need to take an alternative approach when frozen.

This can probably be achieved using MicroHydra's new platform conditional statements, to include different code when frozen.

Simplified example of what I mean:

# mh_if frozen:
# import kanji data from module:
from font import kanji
# mh_else:
# load data on the fly from .txt file
self.font = open("/font/kanji_8x8.txt","r")
# mh_end_if
RealClearwave commented

Since /readme.md and /apps can be generated when .bin file is compiled, I wonder if we can do the same trick and generate /font/kanji_8x8.bin in the filesystem.

echo-lalia commented

Possibly 😊 We could also probably make a separate .py module (with the kanji binary data inside of a const()) to be frozen, and then use the # mh_if frozen:/# mh_end_if macro to switch between using the module/bin file on the regular/frozen versions of the program.

I was planning on trying that out as a potential fix for the next update, but I haven't tested it yet :)

echo-lalia commented

Looks like neither of these options will work (at least, not easily).

The file is too large, and actually can't be frozen into MicroPython with the default partition scheme. I could try expanding the builtin storage, but then we'll lose compatibility with things like M5Launcher (or at least, I'm pretty sure that's the case).

I think the utf8 bin might just have to be a separate/optional/downloadable file, and the program will just have to be designed to work okay without it.

Maybe we could also include some kinda tool in MicroHydra for easily downloading the bin.

RealClearwave commented

What about dumping esp32s3's flash into .bin using esptool.py?

echo-lalia commented

It might be possible to do something like that, but I haven't messed around with that kind of thing a ton, and I think it would have the same issue of incompatibility with M5Launcher.

I'm not sure how I'd like to resolve this, but with the changes I just pushed, utf-8 chars are just drawn as blank spaces when the .bin isn't present.

Off the top of my head, I think it could be cool to extend the terminal app, and maybe include some built-in scripts that could be run for various things, like downloading the utf8 font from the repo. I'm not completely sure, though.

echo-lalia commented

After further investigation, it seems like the new M5Launcher supports firmwares with various partition schemes, so maybe that wont actually be an issue 👀.

I'll still have to figure out how it's possible to include a file like that in the firmware.

Edit: There is an open PR on the MicroPython github for some new enhancements that would allow arbitrary files to be frozen into the rom, and accessed with little RAM overhead. This PR is from 2022, so I'm not holding my breath on it being merged soon, but it'll probably be a perfect solution to this problem once it's ready.

echo-lalia commented

Alright, I did some simple testing.

I converted the entire utf8_8x8 bin into a .py file using the following format:

_UTF8 = const(b'<512KB of font data>')
utf8 = memoryview(_UTF8)

I tried converting it into a .mpy file, and importing it. This gave a memory error, stating that it failed to allocate 512KB.

I tried freezing it into MicroPython along with the other font files like I normally do, but the default partition scheme overflowed.
Finally, I added 1mb to the factory partition (removing 1mb from the vfs partition), and MicroPython compiled successfully.

I tested the import again, and there was no issue! Using micropython.mem _info before and after the import, it seems like it only took about 3760 bytes to import once it was frozen.

So, simply converting it into a .py file for the frozen version of MicroHydra seems like it could work!

  • 🎉1