Reduce memory usage

[ZhymabekRoman]

Hello
Thank you so much for audfprint, your implementation of audio recognition works much better and more accurately than others on Python. I experimented with different settings to achieve an excellent result, and to some extent I managed to find a middle ground, but noticed a very serious problem that others have no implementation - this is a huge use of memory, even with standard settings. From the very beginning I thought that the problem in gzip while saving and opening saved hashes, but as it turned out we really had a problem in pickle, I decided to replace it with HDF5 similar storage using the hickle library (which has an interfeis similar to pickle), but the problem still did not solve it. The use of memory in general when analyzing even 1mb file is approximately 1.5gb, which is very critical (I have Linux with 2.0 GB, Celeron)

What are your thoughts? Could it be better to use MySQL as in this project?

[dpwe]

Do you mean RAM or Disk? Gzip, HDF5 sound like solutions to the space on-disk, but big disks are cheap. I think the RAM usage is more often the problem: the reference table is read into RAM in a space-inefficient (but access speed-efficient) way; by default it occupies 400MB, which I hoped would be OK in current machines. The real problems I've observed are the dynamic memory using when matching against large databases. During matching, the program retrieves lists of all the reference tracks implicated by each individual landmark extracted, then filters to find the most promising reference items. When the reference database gets big, and when the reference items are large, you can end up with a large portion of the reference items looking promising, and so very large initial lists of tracks to check. There's no simple fix here, although shorter reference items can reduce the likelihood of pulling in every item. DAn.

…

On Thu, Jan 28, 2021 at 9:11 AM Zhymabek Roman ***@***.***> wrote: Hello Thank you so much for audfprint, your implementation of audio recognition works much better and more accurately than others on Python. I experimented with different settings to achieve an excellent result, and to some extent I managed to find a middle ground, but noticed a very serious problem that others have no implementation - this is a huge use of memory, even with standard settings. From the very beginning I thought that the problem in gzip while saving and opening saved hashes, but as it turned out we really had a problem in pickle, I decided to replace it with HDF5 similar storage using the hickle <https://github.com/telegraphic/hickle> library (which has an interfeis similar to pickle), but the problem still did not solve it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#81>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEGZUITPJF74UXLPT7WXVTS4FWCJANCNFSM4WXD6QIA> .

[ZhymabekRoman]

I'm sorry I haven't answered for so long.

Do you mean RAM or Disk?

I meant RAM. The files themselves in disk principle inside gzip weigh 1024 times less. I experimented with different methods of saving audio hashes table to a file, and did not achieve much result. The more audio hashes, the more audfprint takes memory during the recognizing. After looking at the code, I came to the conclusion that you need to transfer the mechanism for reading and saving audio hashes table. Now, alas, I'm busy finding time to try to implement what I'm thinking.