Convert from Hash to/from PinnedSizedBytes safely

Question

Convert from Hash to/from PinnedSizedBytes safely

lehins opened this issue 2 years ago · 9 comments

There is a way to implement a total function that does this conversion because we now statically both of their sizes. This came up in #289

Answer 1 · 2022-07-21T21:54:18.000Z

There seems to be an odd distinction between PackedBytes and PinnedSizedBytes at play here. Is there a reason we can't just use the same type for both?

Answer 2 · 2022-07-21T21:55:49.000Z

@kozross They are quite different underneath. PackedBytes are backed by Word64s, while PinnedSizedBytes are backed by an actual ByteArray

Answer 3 · 2022-07-21T22:04:15.000Z

I understand they are implemented quite differently: why can't we unify their representations? I don't really see why they need to be separate in the first place.

Answer 4 · 2022-07-21T22:25:14.000Z

It is right in the name: one is pinned another one is not 😄

Answer 5 · 2022-07-21T22:43:34.000Z

Yeah, I guess that's true, but at the same time, I feel that making both things use ByteArray underneath would make such conversions easier. We could also share APIs in most cases, provided we don't accidentally mix pinned and unpinned things (which these wrappers would prevent).

Answer 6 · 2022-07-21T22:44:03.000Z

I'm also wondering if there'd ever come a time when we might want to generate hashes via the FFI: we don't need this currently, but we might one day.

Answer 7 · 2022-07-21T22:59:34.000Z

I feel that making both things use ByteArray underneath would make such conversions easier.

I see what you are trying to say. Putting pinnedness aside, the whole point why PackedBytes exist is because this:

data Hash32 = Hash32 {-# UNPACK #-} !Word64 {-# UNPACK #-} !Word64 {-# UNPACK #-} !Word64 {-# UNPACK #-} !Word64

will use 30% less memory than

newtype Hash32 = Hash32 (PinnedSizedBytes 32)

We store a lot of hashes in memory and it makes a difference.

I'm also wondering if there'd ever come a time when we might want to generate hashes via the FFI: we don't need this currently, but we might one day.

We are already producing hashes over ffi with libsodium, we just immediately convert them to PackedBytes, so they don't contribute to memory fragmentation and consume less memory, as mentioned above.

Answer 8 · 2022-07-21T23:01:57.000Z

Interesting - I've learned something new today. In that case, the difference in representation definitely makes sense. It would still be nice to have a safe conversion though.

Answer 9 · 2022-07-21T23:04:13.000Z

It would still be nice to have a safe conversion though.

That's exactly what this ticket is about 😉