Bug: Encoding issue when loading hyperbee on browser

[HDegroote]

Summary

I'm running into an encoding issue with Hyperbee on the browser, using hypercore-next. The content arrives in UInt8-arrays instead of the expected Buffer format.

See https://gist.github.com/HDegroote/cf6d3c09a1f07e8dd7b56a8f00168bde for code to replicate. Do note that you need to bundle the code to run it in the browser. I can share bundled code or a webpack config which worked for me if relevant.

Description

I'm using dht-relay to download a hyperbee over hyperswarm. It works as expected on Node, but on the browser the key and value are in Uint8Array format at this point (on Node these are Buffers): https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245

So when the hyperbee-specific decoding is applied, it either crashes (on JSON), or returns something unexpected (e.g. for utf-8).
When running the example in the gist, the output is as expected on Node:

Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)

//Debugging output: printing the values just before executing 
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq:  1
value:  <Buffer 22 76 61 6c 75 65 20 30 22>
key:  <Buffer 62 6c 6f 63 6b 20 23 30>

// Result
block 0:  { seq: 1, key: 'block #0', value: 'value 0' }

When executing on the browser, the code fails:

Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)

//Debugging output: printing the values just before executing 
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq:  1
key: Uint8Array(8) [ 98, 108, 111, 99, 107, 32, 35, 48 ]
value: Uint8Array(9) [ 34, 118, 97, 108, 117, 101, 32, 48, 34 ]

// Crashes when attempting to JSON decode the value:
Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data

The full trace for the hyperbee part is:

Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data
    decodeJSON index.js:52 // JSON lib
    final index.js:248 // This and following are hyperbee line nr's
    get index.js:569
    get index.js:428

Relevant Versions

├─ hypercore@10.0.0-alpha.52
│ ├─ hypercore-crypto@^3.2.1
├─ hyperbee@1.10.1
├─ hyperswarm@4.1.1
│ ├─ @hyperswarm/dht@^6.0.1
├─ @hyperswarm/dht-relay@0.3.0
│ ├─ @hyperswarm/dht@^6.0.1
│ ├─ @hyperswarm/secret-stream@^6.0.0

[mafintosh]

Thanks! Just noting that the uint8s and buffers above are equivalent, so the bug must be in the decoding step.

[HDegroote]

Perhaps that 'Node' valueEncoding on the Hypercore doesn't work well in the browser: https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L356 ? This is where I observe the browser and NodeJS runs begin to differ.

[HDegroote]

As per mafintosh's hint: the issue lies with the codecs library, which on V2.x does not yet use b4a. This is solved by bumping codecs to 3.0.0.

[HDegroote]

On top of bumping the codecs version, a newer version of b4a is also needed to solve this issue. See mafintosh/protocol-buffers-encodings#3

[HDegroote]

Solved with v2.0.0, which bumped the dependencies to the required versions. Thanks!