streamich/memfs

vol.fromJSON(vol.toJSON()) results in corrupt binary file

Steve-Mcl opened this issue · 3 comments

Steve-Mcl commented

Hi, I may be missing something (I've scanned the issues and the help several times) but I cannot see how to store a binary file (like an image), then store the data (using toJSON) for later restore (using fromJSON) without the image becoming corrupt?

My aim is to make a mem fs that can be serialised and saved (as JSON) then at a later time, the volume is restored (fromJSON) and the files should be as they were - however this does not seem to work.

When I write an image to memfs, then immediately read it back - it is a buffer, of the correct size, as expected. However, after dumping the vol toJSON and restoring it fromJSON the image data is corrupt.

Is there a plugin mechanism or hook that I can use to intercept and fix how the buffer is stringified in the toJSON and how it is restored in fromJSON to make memfs a seamless alternative to local fs?

Here is some code that demonstrates the problem.

const {fs: vfs, vol} = require('memfs')
const fs = require('fs')
const { assert } = require('console')
const b64Img = 'iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAApgAAAKYB3X3/OAAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAANCSURBVEiJtZZPbBtFFMZ/M7ubXdtdb1xSFyeilBapySVU8h8OoFaooFSqiihIVIpQBKci6KEg9Q6H9kovIHoCIVQJJCKE1ENFjnAgcaSGC6rEnxBwA04Tx43t2FnvDAfjkNibxgHxnWb2e/u992bee7tCa00YFsffekFY+nUzFtjW0LrvjRXrCDIAaPLlW0nHL0SsZtVoaF98mLrx3pdhOqLtYPHChahZcYYO7KvPFxvRl5XPp1sN3adWiD1ZAqD6XYK1b/dvE5IWryTt2udLFedwc1+9kLp+vbbpoDh+6TklxBeAi9TL0taeWpdmZzQDry0AcO+jQ12RyohqqoYoo8RDwJrU+qXkjWtfi8Xxt58BdQuwQs9qC/afLwCw8tnQbqYAPsgxE1S6F3EAIXux2oQFKm0ihMsOF71dHYx+f3NND68ghCu1YIoePPQN1pGRABkJ6Bus96CutRZMydTl+TvuiRW1m3n0eDl0vRPcEysqdXn+jsQPsrHMquGeXEaY4Yk4wxWcY5V/9scqOMOVUFthatyTy8QyqwZ+kDURKoMWxNKr2EeqVKcTNOajqKoBgOE28U4tdQl5p5bwCw7BWquaZSzAPlwjlithJtp3pTImSqQRrb2Z8PHGigD4RZuNX6JYj6wj7O4TFLbCO/Mn/m8R+h6rYSUb3ekokRY6f/YukArN979jcW+V/S8g0eT/N3VN3kTqWbQ428m9/8k0P/1aIhF36PccEl6EhOcAUCrXKZXXWS3XKd2vc/TRBG9O5ELC17MmWubD2nKhUKZa26Ba2+D3P+4/MNCFwg59oWVeYhkzgN/JDR8deKBoD7Y+ljEjGZ0sosXVTvbc6RHirr2reNy1OXd6pJsQ+gqjk8VWFYmHrwBzW/n+uMPFiRwHB2I7ih8ciHFxIkd/3Omk5tCDV1t+2nNu5sxxpDFNx+huNhVT3/zMDz8usXC3ddaHBj1GHj/As08fwTS7Kt1HBTmyN29vdwAw+/wbwLVOJ3uAD1wi/dUH7Qei66PfyuRj4Ik9is+hglfbkbfR3cnZm7chlUWLdwmprtCohX4HUtlOcQjLYCu+fzGJH2QRKvP3UNz8bWk1qMxjGTOMThZ3kvgLI5AzFfo379UAAAAASUVORK5CYII='

// cleanup
try { fs.unlinkSync('fsImg.png') } catch (error) { }

// init vol
vol.fromJSON({}, '/')

// Read image from local fs
const img = Buffer.from(b64Img, 'base64')

// write img to local fs, read from local fs & check 
fs.writeFileSync('fsImg.png', img)
const fs_img = fs.readFileSync('fsImg.png')
assert(fs_img.length === img.length, `fs_img.length (${fs_img.length}) === (${img.length}) img.length`) // OK


// write img to mem fs, read from mem fs & check
vol.writeFileSync('/vfsImg.png', img)
const vfsImg = vol.readFileSync('/vfsImg.png')
assert(vfsImg.length === img.length, `vfsImg.length (${vfsImg.length}) === (${img.length}) img.length`) // OK

// serialize mem fs for later restore
const volDump = vol.toJSON();

// clear vol
vol.reset();

// restore mem fs
vol.fromJSON(volDump)

// re-read img from mem fs & check
const vfsImg2 = vol.readFileSync('/vfsImg.png')
assert(vfsImg2.length === img.length, `vfsImg2.length (${vfsImg2.length}) === (${img.length}) img.length`) // NOT OK!
streamich commented

The Volume built-in .toJSON and .fromJSON utilities support only text data. In v4 release there will be snapshot utility, which supports binary data. Try installing v4, here are the docs: https://github.com/streamich/memfs/blob/next/docs/snapshot/index.md

npm i memfs@next
  • 👍1
Steve-Mcl commented

Thanks. I'll give it a go.

Out of curiosity, is this v4 release imminent? Is the beta considered feature complete?

streamich commented

@Steve-Mcl Planning to release v4 later today.

  • 👍1