Bug: Encoding issue when loading hyperbee on browser
Closed this issue · 5 comments
Summary
I'm running into an encoding issue with Hyperbee on the browser, using hypercore-next. The content arrives in UInt8-arrays instead of the expected Buffer format.
See https://gist.github.com/HDegroote/cf6d3c09a1f07e8dd7b56a8f00168bde for code to replicate. Do note that you need to bundle the code to run it in the browser. I can share bundled code or a webpack config which worked for me if relevant.
Description
I'm using dht-relay to download a hyperbee over hyperswarm. It works as expected on Node, but on the browser the key and value are in Uint8Array format at this point (on Node these are Buffers): https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
So when the hyperbee-specific decoding is applied, it either crashes (on JSON), or returns something unexpected (e.g. for utf-8).
When running the example in the gist, the output is as expected on Node:
Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)
//Debugging output: printing the values just before executing
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq: 1
value: <Buffer 22 76 61 6c 75 65 20 30 22>
key: <Buffer 62 6c 6f 63 6b 20 23 30>
// Result
block 0: { seq: 1, key: 'block #0', value: 'value 0' }
When executing on the browser, the code fails:
Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)
//Debugging output: printing the values just before executing
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq: 1
key: Uint8Array(8) [ 98, 108, 111, 99, 107, 32, 35, 48 ]
value: Uint8Array(9) [ 34, 118, 97, 108, 117, 101, 32, 48, 34 ]
// Crashes when attempting to JSON decode the value:
Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data
The full trace for the hyperbee part is:
Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data
decodeJSON index.js:52 // JSON lib
final index.js:248 // This and following are hyperbee line nr's
get index.js:569
get index.js:428
Relevant Versions
├─ hypercore@10.0.0-alpha.52
│ ├─ hypercore-crypto@^3.2.1
├─ hyperbee@1.10.1
├─ hyperswarm@4.1.1
│ ├─ @hyperswarm/dht@^6.0.1
├─ @hyperswarm/dht-relay@0.3.0
│ ├─ @hyperswarm/dht@^6.0.1
│ ├─ @hyperswarm/secret-stream@^6.0.0
Thanks! Just noting that the uint8s and buffers above are equivalent, so the bug must be in the decoding step.
Perhaps that 'Node' valueEncoding on the Hypercore doesn't work well in the browser: https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L356 ? This is where I observe the browser and NodeJS runs begin to differ.
As per mafintosh's hint: the issue lies with the codecs library, which on V2.x does not yet use b4a. This is solved by bumping codecs to 3.0.0.
On top of bumping the codecs version, a newer version of b4a is also needed to solve this issue. See mafintosh/protocol-buffers-encodings#3
Solved with v2.0.0, which bumped the dependencies to the required versions. Thanks!