Streaming cache memory leak

[liampauling]

Cache leaks memory due to markets/orders not being cleared on closure, more info here.

[synapticarbors]

Yeah, I overrode the _process method of the MarketStream to delete markets 18 hours after their last update and do something similar with the OrderStream:

    def _process(self, market_books, publish_time):
        super()._process(market_books, publish_time)

        now = datetime.datetime.utcnow()

        for market_book in market_books:
            mid = market_book['id']
            self.market_last_update[mid] = now

        if (now - self.cache_last_check).total_seconds() > 600:
            for mid in list(self._caches.keys()):
                if (now - self.market_last_update[mid]).total_seconds() > 64800:
                    del self._caches[mid]
                    logger.info('Market {} removed from market cache, {} markets in cache'.format(
                        mid, len(self._caches)))

            self.cache_last_check = now

It looks like Flumine is removing markets on closure, so unless this isn't working or I'm misunderstanding the code, I'm not sure why you were seeing the issue with Flumine:

https://github.com/liampauling/flumine/blob/fd26d26947a9cc0ab0f71102f4081415ad5fed4c/flumine/streams/datastream.py#L52-L58

[liampauling]

Yeah flumine does it for the data stream but that is only when recording data, anything live isn't, I have a similar idea to above and I think 18-24 hours should reduce the risk to near enough 0.