SkeLLLa/node-object-hash

hasher throws out of memory exception on very large Uint8array instances crashing node

Opened this issue · 2 comments

wayfarer3130 commented

Trying to hash a 256 mb Uint8array crashes the node instance with an out of memory. It appears the base object is just too large to be handled this way. It would be nice to not convert array buffers into strings, but hash them raw.

SkeLLLa commented

Hi. Good catch. Would you like to investigate it and make a PR?

wayfarer3130 commented

It is a significant change to do this correctly - the right way of doing this is to provide hash generator to the child nodes so that you never create a full string, but do it piecemeal. The issue with that is it is quite different in design from the current behaviour. That would make it quite a bit faster, because you wouldn't be combining/moving strings in large numbers the way it does now, but trying to do that the way it is now would be hard.

There is a pair of simple changes that would "fix" things - that is, would make them reasonably fast, without changing the structure much:

  1. If the incoming object is an ArrayBuffer type, then return the hash code directly
  2. For the "toString" method on values, return the hash code of ArrayBuffer instances rather than a string representation

Those two changes would allow either simple objects or things contains arrays to be hashed at a reasonable speed.