DataDog/datadog-go

client.Flush() doesn't flush synchronously

Closed this issue · 4 comments

Hi.
I'm facing a strange behavior in Flush(). When I run the code below, it doesn't finish flushing then results in deadlock failure.
https://play.golang.org/p/90sBE2ed9dV
Can someone help?

arbll commented

Hi @rerorero,
There was indeed a bug with custom writers that would lead to the majority of the metrics being dropped, fixing it in #106.

Regarding your test

if buf.Len() == 0 {

would never be true after the flush. This buffer represents the output of the client in your scenario so a flush should result in this buffer containing 3000 formatted metrics, not being empty.
That said, because of the bug above a good chunk of those 3000 metrics would get dropped. Once it's fixed you should get them all in buf.

LMK if you have questions,
Arthur

@arbll thank you for reply! sorry, I had a mistake in the code. buf.Len() == 0 is wrong.

But even after fixing my code, I still see an another strange things.

With this code looks like Flush() doesn't flush synchronously. Is this expected?

I checked #106 but it doesn't affect the code because I don't pass any options and resolveOptions would set the default values.

arbll commented

@rerorero this library starts dropping metrics when it's inner queues are full. This might seem surprising but this is a requirement to ensure that the app making calls to this library won't ever be blocked for a long time.

The bug in #106 leads to extremely small queues (1 element) when you are using custom writers (not udp/uds) leading to metrics being dropped very easily.

This is most likely why you don't have all the metrics you are expecting in the final buffer after the flush.

I see, thank you for kind explanation.