segmentio/encoding

Proposal: ability to marshal with flags

Jerska opened this issue · 3 comments

Hi! Me again with another proposal!

json.Parse offers a readily available alternative to json.Unmarshal with the simple addition of flags.

In comparison, json.Append takes two extra parameters compared to json.Marshal: a []byte to fill and the flags.
This means that unfortunately, you can't benefit from the encoderBufferPool if you just want to add some AppendFlags.

This would give 3 functions from most to least flexible :

  • Append(b []byte, x interface {}, flags AppendFlags) - can use tags, no extra copy, but no buffer pool
  • MarshalWithFlags(x interface{}, flags AppendFlags) - can use tags, buffer pool & one extra copy
  • Marshal(x interface{}) - cannot use tags, buffer pool & one extra copy

Having to pre-allocate a buffer for Append might be quite a challenge, because you often don't know how long the resulting buffer will be.
For instance, allocating a full page like the buffer pool does is not necessarily a great option for small payloads.
Reimplementing a similar pool in our code is definitely an option, but i'd be great to simply be able to reuse the logic that's already there in the lib.

Would this be something you'd consider adding in? Again, happy to propose a PR.

Just to make sure someone sees this, cc @achille-roussel .

I'd understand that you don't want to modify the API too much, feel free to close if you don't believe this would be of interest.

Thanks for bumping this in my inbox!

If I'm understanding correctly, the core issue is that we cannot leverage the encoder buffer pool in json.Append?

I'm curious if you have used a workaround in your application today?

Thank you for your response!

If I'm understanding correctly, the core issue is that we cannot leverage the encoder buffer pool in json.Append?

Exactly !

I'm curious if you have used a workaround in your application today?

I've managed to mostly use json.Append for things where I could approximate the buffer size I needed to pre-allocate.

However, for a few others, I've resorted to using json.Marshal.
For those structures the benefits of the preallocated buffer are greater than the optimizations provided by removing SortKeys and EscapeHTML.
It would be more optimized for me to manually handle a buffer pool on our side and use json.Append, but this kind of additions wouldn't fit our criteria for complexity / perf improvements.