streamer dropping events

[KiriakosGeorgiou]

I'm testing the streamer against mine (which is written in ruby with eventmachine, has been proven to work well for two years) and I noticed delays. I started digging into it more comparing every single trade for a specific low volume symbol, and noticed trade events are being dropped. I am streaming 150 symbols, so I suspect something must not be right related to goroutines. I used https://github.com/bcicen/grmon to monitor goroutines, and sure enough I didn't see as many goroutines as I'd expect for the streamer to keep up with the fast and furious pace of trades coming in. I'm a go newbie so I'm still digging around about this, but I thought to let you know.

Shouldn't a new goroutine be spawned for every incoming line? otherwise the streamer can't keep up.

[timpalpant]

You can create different MarketEventStreams for subsets of symbols. Each streamer is handled by a single goroutine, so you can choose how many you want but need to manually shard over symbols. Creating a goroutine for every line/event will have too much overhead. In my experience a single goroutine could keep up with a few hundred symbols, but it will depend on the hardware.

You should create the output events channel with sufficient buffer to be able to handle spikes. You also want to dequeue from the output events channel as fast as possible, and can do additional buffering if needed at this point to make sure that it does not overflow. You may also want to have multiple goroutines consuming from the events channel to process them in parallel.