Memory leak in _speedups.c ?

[eug3nix]

Hi.

We are using Genshi template engine in one of our projects.
After we upgraded to the latest stable Genshi release (we were using version 0.7.0 before) we have noticed what seems to be a memory leak.

Here is a minimal application which shows that behavior.
memleak.tar.gz

I was able to isolate the exact commit where this issue started to show up:
99ad51d

I am not sure what exactly causes the issue but the memory leak is gone if we either:
A)remove _speedupds.so
B)revert to a Genshi version before the commit mentioned above.

Here are the lisf of steps to reproduce the issue:
0. install the project dependencies and run the app

1. observe the process memory usage before the test

$ ps axwu|grep memleak.py|grep -v grep  | awk '{print $5}'
986740

2. make requests to the sample app

$ ab -v0 -n10000 -c10 http://localhost:8080/ > /dev/null 2>&1

3. check the memory usage again

$ ps axwu|grep memleak.py|grep -v grep  | awk '{print $5}'
994932

Note: the memory usage is constantly growing with each subsequent run of the test

Environment details:

$ pip list

Package                       Version
----------------------------- -------
backports.functools-lru-cache 1.6.4
cheroot                       8.5.2
CherryPy                      17.0.0
FormEncode                    2.0.0
Genshi                        0.7.5
jaraco.functools              2.0
more-itertools                5.0.0
pip                           20.3.4
portend                       2.6
pytz                          2021.1
selectors2                    2.0.2
setuptools                    45.0.0
six                           1.16.0
tempora                       1.14.1
wheel                         0.37.0
zc.lockfile                   2.0

$ python -V
Python 2.7.17

Sincerely,
Eugene

[hodgestar]

Thank you for a very nice bug report. Could I ask you to try reducing the test case one step more to a smaller one that uses just Markup(...).join(...) and perhaps uses gc.collect() and len(gc.get_objects()) to check the reference count. There is quite a lot going on in your example (13 dependencies, a web server) and I couldn't see anything obviously wrong in 99ad51d.

[eug3nix]

Thank you for your feedback.
Here is the adjusted test and the results I have got from it

test.py.txt

$ pip list |grep Genshi && python ./test.py

Genshi                        0.7.5
objcount before: 5053
RSS before: 8692
objcount after: 5054
RSS after: 294336

$ pip install Genshi==0.7.0

$ pip list |grep Genshi && python ./test.py
Genshi                        0.7
objcount before: 4166
RSS before: 7988
objcount after: 4167
RSS after: 39784

[hodgestar]

Thanks! Fix created in #47.

[hodgestar]

Fix merged.