msgpack/msgpack-python

memory leak observed in python 3.12.1

TheRealMcoy opened this issue · 5 comments

TheRealMcoy commented

This library appears to leak memory in Unpacker method under Python 3.12.1

reproducer script:

import msgpack
from io import BytesIO

while True:

   buf = BytesIO()
   for i in range(100):
      buf.write(msgpack.packb(i, use_bin_type=True))

   buf.seek(0)

   unpacker = msgpack.Unpacker(buf, raw=False)
   for unpacked in unpacker:
       print(unpacked)

Python 3.11.7 after 10 minute run:
python3 11

Python 3.12.1 after 10 minute run:
python3 12

senyai commented

I have similar situation, but my test case is a bit different:

from msgpack import Unpacker, packb

unpacker = Unpacker()
network_packet = packb(None)

for i in range(2000000):
    unpacker.feed(network_packet)
    for unpacked in unpacker:
        pass
  • 👍1
methane commented

@TheRealMcoy @senyai do you use macOS?

methane commented

Sorry, OS is not related.

cython/cython#5724 caused it.
Until I make mew release, manually run Cython locally will fix it.

  • 👍2
senyai commented

@methane recompiling with new cython fixed it for me, much appreciated.

TheRealMcoy commented

@methane the latest 1.0.8 has resolved the memory leak, many thanks!

  • 🎉1