Level/community

Dump Buffer for TypedArrays (for compactness and efficiency)

SilentCicero opened this issue · 6 comments

SilentCicero commented

Hi all,

Coming from the browser focused world, I see many of the Level libraries don't really use a ton of Buffer facility, and considering Buffer's can be easily fitted (and are just a view to) typed arrays, why not just dump the massive Buffer library altogether.

This would make Level a much leaner construction for browser based or p2p apps like mine, and may even make processing at a micro level faster.

Thoughts?

vweevers commented

Some nuance: buffer is 147 bytes minified and gzipped. Calling it "massive" isn't fair, especially considering the interoperability that it enables. That said, I understand the need to shave off bytes wherever possible.

Unfortunately, the level ecosystem does use buffer heavily. The dozens of modules probably only use a small part of buffer's API, that's true, but switching to typed arrays (or restricting ourselves to UInt8Array's API) is nevertheless a huge breaking change and a huge effort.

vweevers commented

I'd like to keep this open as a reminder. And to allow folks to chime in; I don't want to give the signal "it's never gonna happen".

SilentCicero commented
MeirionHughes commented

I'm not even sure you'd save anything in the long run - Buffer gives extensive encoder support - that would have to be moved in level* if its not on the Buffer.

If its something you're passionate about, I'd recommend forking abstract-leveldown and try passing a Buffer prototype through to its constructor, which would be used in-place of require'ing the existing Buffer ponyfill. doing it this way is the least painful approach I can think of - but give the user the option to pass a custom cut-down prototype explicit for their needs and stack.

I'm more concerned with multiple Buffer ponyfills more than anything - I'm pretty sure current build (webpack) is bundling another Buffer polyfill. Also better support for Uint8Array would be nice: in my custom encoder I have to wrap a Uint8Array in a Buffer else its re-encoded, for some reason.

vweevers commented

I'm not even sure you'd save anything in the long run - Buffer gives extensive encoder support - that would have to be moved in level* if its not on the Buffer.

The answer for that is to not use level, which is a convenience module that is unlikely to drop buffer. Folks that don't want buffer should instead do something like:

const compose = require('level-compose')
const factory = compose(leveldown, levelup)
const db = factory('./db')

Which is to say, those folks won't get encodings, unless we write an alternative to encoding-down (see below). Down the line, once abstract-leveldown reaches feature parity with levelup, folks can use leveldown (or other) directly.

WIP list of tasks to make buffer optional (feel free to edit):

  • Replace Buffer.isBuffer with is-buffer (Level/abstract-leveldown@8f79234)
  • Determine what to do with asBuffer / keyAsBuffer / valueAsBuffer
  • Add db.supports.buffer (level-supports) and use that in abstract tests
  • Write a new module that, like level, exports either leveldown or level-js depending on environment. To get the same convenience as level but without buffer.
  • Write a new module similar to encoding-down but with bring-your-own codecs
  • Determine what to do in memdown, or write an alternative that is backed by typed arrays
  • Determine what to do in level-js, oh boy
  • 👍1
vweevers commented

Covered by #102 and https://github.com/Level/transcoder. It will be possible to use either Uint8Array, Buffer, or both, and to exclude the buffer shim from browser bundles if you choose to use Uint8Array.