Reduce memory usage for text creation: bitmap_label

Question

Reduce memory usage for text creation: bitmap_label

kmatch98 opened this issue 4 years ago · 9 comments

In order to reduce the amount of memory for displaying text, @FoamyGuy and I are developing a new "label" class that will use a bitmap instead of a group of TileGrids to store the text. Here is a proposed strategy for this new "bitmap_label" Class:

`bitmap_label` Class definition

Input parameters:
- Make text a required parameter (Note: This will flag if someone creates a bitmap_label with no text, which doesn’t make any sense.)
- For direct compatibility with label, keep the max_glyphs parameter, but ignore it.
Do we want to keep self._memorySaver? (Note: If so, update this to a class variable)
- True - do not save the text value
- False - save the text value
- If we keep this option, then we need to create a getter for text, returns None if self._memorySaver == True.
Since the bitmap_label has limited edit capability after creation, minimize the instance variables that are stored
Add getters/setters for:
- anchor_point
- anchor_position
- color
- background_color
- bounding_box (x,y, w,h)
Getters only (or does it regenerate from scratch if this is changed?)
- font
- line_spacing
- text (returns None if text is not stored with _memory_saver option)
Add scale capability (this should be handled in the Group)

General steps for `_init_` of a bitmap_label instance:

Error-checking: If text is “”, raise an exception.
Calculate the text box size, including any newlines and tabs
Calculate the background box size, including padding
Note: Negative padding cannot be used. (This would require adding a third color to the palette, which would inflate the bitmap_label bitmap memory usage.)
Determine the bitmap size required by comparing the text box and background box max x,y dimensions.
Determine start location of text in the bitmap
Initialize the palette, TileGrid (self), Bitmap
Draw the background fill, as required
Place the text in the bitmap
Set the TileGrid x,y placement based on anchor_point and anchor_position.
Updated bounding_box values.

Answer 1 · 2020-07-28T18:59:20.000Z

Is the core problem with the current implementation the overhead of the TileGrid class? How does this change as the text gets larger? The TG model saves glyph bitmap duplication at the expense of the TG fixed overhead and the cost of storing it in a group.

Note, I wouldn't include the current background bitmap in the calculations because it's easy to optimize on it's own.

Answer 2 · 2020-07-28T22:32:30.000Z

Short answer to your question:

Conclusion: `label.py` uses ~100 bytes per character for a 16pt font.

Here are the details

Per your questions, I did a trial between label.py and my TextMap library (https://github.com/kmatch98/CircuitPython_textMap).

For this trial, I used a string (659 characters long). I printed one additional character during each loop and then measured the memory after each 10 characters are added. I compared two fonts (Helvetica Bold 16 and Vera Sans Roman 24).

For your reference, my TextMap example creates a bitmap the size of the screen (320x240 in this case) and copies each glyph's bits into the bitmap.

I measured:

Memory available before starting the loop of printing the characters (to estimate "overhead"), using gc.collect() then gc.mem_free()
Memory reduction after each character is added (bytes per added character)

Sample text output from `label.py` usage with Helvetica Bold 16:

After display.show(myGroup), just before loop start  Memory free: 85232
charCount: 10, Running Memory free: 84208, mem usage in Loop: 1024, per character mem: 93.0909
charCount: 20, Running Memory free: 82944, mem usage in Loop: 2288, per character mem: 108.952
charCount: 30, Running Memory free: 81856, mem usage in Loop: 3376, per character mem: 108.903
charCount: 40, Running Memory free: 80656, mem usage in Loop: 4576, per character mem: 111.61
charCount: 50, Running Memory free: 79536, mem usage in Loop: 5696, per character mem: 111.686
charCount: 60, Running Memory free: 78368, mem usage in Loop: 6864, per character mem: 112.525
charCount: 70, Running Memory free: 77232, mem usage in Loop: 8000, per character mem: 112.676

Sample text output from `TextMap` usage with Helvetica Bold 16:

After display.show(myGroup), just before loop start  Memory free: 86192
charCount: 10, Running Memory free: 86320, mem usage in Loop: -128, per character mem: -11.6364
charCount: 20, Running Memory free: 86192, mem usage in Loop: 0, per character mem: 0.0
charCount: 30, Running Memory free: 86192, mem usage in Loop: 0, per character mem: 0.0
charCount: 40, Running Memory free: 86192, mem usage in Loop: 0, per character mem: 0.0
charCount: 50, Running Memory free: 86192, mem usage in Loop: 0, per character mem: 0.0

Experimental setup:

Itsy Bitsy NRF52840
ILI9341 Display 320x240

Tabular Results:

Summary of results:

For label.py, each added character increases memory usage by 110 to 125 bytes per character (dependent upon font size).
For a large string (659 characters), the label.py has the largest overhead.
Changing max_glyphs of 659 to 30 reduces the memory overhead by 5 kB, and then label.py has less overhead.

Conclusion: Using a lot of text at ~100 bytes per character eats up memory in a hurry.

Answer 3 · 2020-07-29T18:14:36.000Z

@kmatch98 Thank you for the empirical numbers!

I think the textmap numbers are a bit misleading though because it's a fixed overhead of 320 x 240 // 8 = 9600 bytes which means label is a better choice for strings of 9600 // 125 = 76 characters or less.

We can raise this number by reducing the overhead of each tilegrid. I think the cost of each character now is:

4 bytes for the Group entry pointer.
the size of the displayio_tilegrid_obj_t
the size of the glyph's bitmap

Only the first two bullets apply when a duplicate character is used.

I think we should add a TextArea class to this library (in a different file) that takes the approach you suggest. It can optimize for multi-line text areas (aka paragraphs) while label can focus on short text, usually in a single line.

Answer 4 · 2020-07-29T18:33:39.000Z

I reviewed my code and decided to eliminate any difference due to the import statements. So now the import statements are consistent for all tests. Here is chart with the updated overhead:

@tannewt To check your comment about the third bullet, reran the code with a string with just repeated "M" in it.

I see the same ~100 byte incremental memory usage when each character is added, even thought it is a duplicated glyph. Am I missing something?

Please note that I load the glyphs before entering the loop:

glyphs = b'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-,.:?! '
font.load_glyphs(glyphs)`

As for overhead, the updated numbers show that label.py overhead is 5-10kB less, as you said.

Raw data and Test Code for `label.py` with "M" string

Here is the output result with the repeated "M" string using label.py and Helvetica 16:

code.py output:
Starting the display...
spi.frequency: 32000000
Display is started
loading fonts...
loading glyphs...
Glyphs are loaded.
Fonts completed loading.
After creating Group,  Memory free: 91520
myString: Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box.
string length: 659
After display.show(myGroup), just before loop start  Memory free: 85808
charCount: 10, Running Memory free: 84160, mem usage in Loop: 1648, per character mem: 149.818
charCount: 20, Running Memory free: 82912, mem usage in Loop: 2896, per character mem: 137.905
charCount: 30, Running Memory free: 81824, mem usage in Loop: 3984, per character mem: 128.516
charCount: 40, Running Memory free: 80640, mem usage in Loop: 5168, per character mem: 126.049
charCount: 50, Running Memory free: 79536, mem usage in Loop: 6272, per character mem: 122.98
charCount: 60, Running Memory free: 78368, mem usage in Loop: 7440, per character mem: 121.967
charCount: 70, Running Memory free: 77248, mem usage in Loop: 8560, per character mem: 120.563
charCount: 80, Running Memory free: 76096, mem usage in Loop: 9712, per character mem: 119.901
charCount: 90, Running Memory free: 74976, mem usage in Loop: 10832, per character mem: 119.033
charCount: 100, Running Memory free: 73872, mem usage in Loop: 11936, per character mem: 118.178
charCount: 110, Running Memory free: 72704, mem usage in Loop: 13104, per character mem: 118.054
charCount: 120, Running Memory free: 71584, mem usage in Loop: 14224, per character mem: 117.554
charCount: 130, Running Memory free: 70480, mem usage in Loop: 15328, per character mem: 117.008
charCount: 140, Running Memory free: 69312, mem usage in Loop: 16496, per character mem: 116.993
charCount: 150, Running Memory free: 68208, mem usage in Loop: 17600, per character mem: 116.556

For reference, my full code for memory usage testing of label.py with the "M" string is shown below:

# Sample code using the textMap library and the "textBox" wrapper class
# Creates four textBox instances
# Inserts each textBox into a tileGrid group
# Writes text into the box one character at a time
# Moves the position of the textBox around the display
# Clears each textBox after the full string is written (even if the text is outside of the box)

import textmap
from textmap import textBox

import board
import displayio
import time
import terminalio
import fontio
import sys
import busio
#from adafruit_st7789 import ST7789
from adafruit_ili9341 import ILI9341
from adafruit_display_text import label

#  Setup the SPI display

print('Starting the display...') # goes to serial only
displayio.release_displays()


spi = board.SPI()
tft_cs = board.D9 # arbitrary, pin not used
tft_dc = board.D10
tft_backlight = board.D12
tft_reset=board.D11

while not spi.try_lock():
    spi.configure(baudrate=32000000)
    pass
spi.unlock()

display_bus = displayio.FourWire(
    spi,
    command=tft_dc,
    chip_select=tft_cs,
    reset=tft_reset,
    baudrate=32000000,
    polarity=1,
    phase=1,
)

print('spi.frequency: {}'.format(spi.frequency))

DISPLAY_WIDTH=320
DISPLAY_HEIGHT=240

#display = ST7789(display_bus, width=240, height=240, rotation=0, rowstart=80, colstart=0)
display = ILI9341(display_bus, width=DISPLAY_WIDTH, height=DISPLAY_HEIGHT, rotation=180, auto_refresh=True)

display.show(None)

print ('Display is started')


# load all the fonts
print('loading fonts...')

import terminalio


fontList = []
fontHeight = []

##### the BuiltinFont terminalio.FONT has a different return strategy for get_glyphs and 
# is currently not handled by these functions.
#fontList.append(terminalio.FONT)
#fontHeight = [10] # somehow the terminalio.FONT needs to be adjusted to 10

# Load some proportional fonts
fontFiles =   [
            'fonts/Helvetica-Bold-16.bdf',
            'fonts/BitstreamVeraSans-Roman-24.bdf', # Header2
            'fonts/BitstreamVeraSans-Roman-16.bdf', # mainText
            ]

from adafruit_bitmap_font import bitmap_font

for i, fontFile in enumerate(fontFiles):
    thisFont = bitmap_font.load_font(fontFile) 
    fontList.append(thisFont)
    fontHeight.append( thisFont.get_glyph(ord("M")).height ) 

preloadTheGlyphs= True # set this to True if you want to preload the font glyphs into memory
    # preloading the glyphs will help speed up the rendering of text but will use more RAM

if preloadTheGlyphs:

    # identify the glyphs to load into memory -> increases rendering speed
    glyphs = b'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-,.:?! '

    print('loading glyphs...')
    for font in fontList:
        font.load_glyphs(glyphs)
    print('Glyphs are loaded.')

print('Fonts completed loading.')

# create group 
import gc

gc.collect()
myGroup = displayio.Group( max_size=1 ) # Create a group for displaying
tileGridList=[] # list of tileGrids
print( 'After creating Group,  Memory free: {}'.format(gc.mem_free()) )

gc.collect()

myString=('Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box. Full Screen Size: This is a stationary box, not a stationery box.')
#myString=('MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM')
print('myString: {}'.format(myString))
print('string length: {}'.format(len(myString)))

text_label = label.Label(font=fontList[0], text="", color=0xFFFFFF, max_glyphs=len(myString))
myGroup.append(text_label)

display.show(myGroup)

charCount=0

gc.collect()
memBeforeLoop=gc.mem_free()
print('After display.show(myGroup), just before loop start  Memory free: {}'.format(memBeforeLoop) )

while True:

    # Add characters one at a time.

    if charCount >= len(myString):
        charCount=0
    text_label.text=myString[0:charCount] # add a character

    
    charCount += 1

    # Print the memory availability every 10 movements.
    if charCount % 10 == 0:
        gc.collect()
        currentMem=gc.mem_free()
        print( 'charCount: {}, Running Memory free: {}, mem usage in Loop: {}, per character mem: {}'.format(charCount, currentMem, memBeforeLoop-currentMem, (memBeforeLoop-currentMem)/(charCount+1)) )

Answer 5 · 2020-07-29T18:54:25.000Z

To keep from getting lost in the empirical measurements:

Once we better define what the problem is, I want to make sure we have an agreed strategy before going any further.

I see two approaches to consider:

Make a drop-in replacement for label.py, only difference is that it's in a bitmap (and less mutable on the fly). This is the suggestion that was discussed in the call on July 20th.
Make a library that is good for rendering larger areas of text, possibly including other features (e.g. word wrapping, cursor control like a terminal)

Based on what I have seen the current label.py uses an unexpectedly large amount of memory even for small amounts of text (> 76 chars?) and is causing people problems. If that's not really a problem then path #2 is probably is a better approach, to focus on applications specifically that are text-heavy. However, text-heavy applications may prefer "mutable" structures so we'll have to consider how to respond to that if path #2 is preferred.

Y'all's feedback is welcome, requested and appreciated! @FoamyGuy @tannewt

Answer 6 · 2020-07-30T18:06:15.000Z

I see the same ~100 byte incremental memory usage when each character is added, even thought it is a duplicated glyph. Am I missing something?

Nope! I didn't realize you were preloading the glyphs.

I see two approaches to consider:

Make a drop-in replacement for label.py, only difference is that it's in a bitmap (and less mutable on the fly). This is the suggestion that was discussed in the call on July 20th.

Make a library that is good for rendering larger areas of text, possibly including other features (e.g. word wrapping, cursor control like a terminal)

I would do more 2 than 1. To me, a label is a small amount of text. Hundreds of characters is not a label but rather a paragraph or text area. I don't think it needs to be in a different library though. It should just be a different module (aka file) in this library.

Based on what I have seen the current label.py uses an unexpectedly large amount of memory even for small amounts of text (> 76 chars?) and is causing people problems. If that's not really a problem then path #2 is probably is a better approach, to focus on applications specifically that are text-heavy. However, text-heavy applications may prefer "mutable" structures so we'll have to consider how to respond to that if path #2 is preferred.

I agree that 100 bytes per character is a lot. We should take a look at TileGrid to see if we can reduce that overhead. I don't think that is the primary concern of this investigation though because the memory errors people hit are due to large consecutive allocations because of the group containing many characters. 100 bytes per TG is a lot but at least they are separate memory chunks.

I'd propose adding a TextArea class that takes rectangular bounds and allocates a bitmap for the bounds once up front. Then, text can be laid out into it repeatedly without additional allocations.

Y'all's feedback is welcome, requested and appreciated! @FoamyGuy @tannewt

Thank you for your in depth thoughts on this!

Answer 7 · 2020-07-31T02:01:23.000Z

Thanks @tannewt. I remain suspicious of TileGrid, it seems like the overhead it excessive but I can't understand the code well enough to point to the issue.

After good discussion with @FoamyGuy this morning I decided to start with the direct drop-in replacement bitmap_label (type #1 above).

Once that is done, I'll start on #2. I've already been down this road with TextMap, so those features will be my starting scope. Main scope is to decide what features to bring from recent updates to label.py.

But before I turn to #2 , I'd like elicit feedback on the top features desired. I'm not sure the best forum to do that. I suspect the easiest method is to create something and then people will say "I wish..." The In-the-Weeds time is helpful, but I'm unsure if we can collect specific text-realted feedback there.

I am totally new to this kind of large-scale collaborative development, so I'm open to your suggestions how to scope a new add-on file to the display_text library.

Answer 8 · 2020-07-31T17:53:15.000Z

I think the first place to look for overhead is here: https://github.com/adafruit/circuitpython/blob/main/shared-module/displayio/TileGrid.h#L37

Most of it could be simplified for the single tile case and we could therefore split much of it into a separate sub-allocation when more than one tile is used.

Answer 9 · 2020-08-18T00:12:31.000Z

Closed with: #79

bitmap_label Class definition

General steps for _init_ of a bitmap_label instance:

Conclusion: label.py uses ~100 bytes per character for a 16pt font.