Sia - Binary serialisation and deserialisation with built-in compression. You can consider Sia a strongly typed, statically typed domain specific binary language for constructing data. Sia preserves data types and supports custom ones.
Please note the Sia specification and implementation isn't final yet. As a core part of Clio programming language, Sia evolves with Clio. It is made to make fast RPC calls possible.
I needed a fast schema-less serialization library that preserves type info and is able to code/decode custom types. I couldn't find one. At first I wanted to go with a JSON with types solution but it didn't work out, so I created my own.
This repository contains a pure JS implementation of Sia, on our test data Sia is 66% to 1250% faster than JSON and serialized data (including type information for all entries) is 10% to 30% smaller than JSON. Sia is faster and smaller than MessagePack and CBOR/CBOR-X. It is possible to use lz4 to compress Sia generated data even more and still be faster than JSON, MessagePack and CBOR-X.
Tests are run on a 2.4 GHz 8-Core Intel Core i9-9980HK CPU (5 GHz while running the benchmarks)
with 64 GB 2667 MHz DDR4 RAM. Node version 16.4.2, Mac OS 11.5.1. 100 loops each serialization library.
To run the benchmark suite you can run npm run benchmark
and to run the tests you can run npm run test
.
Read specs.md.
To install the Node library and save it as a dependency, do:
npm i -S sializer
WIP
The Node Sia library exports 5 items:
const { sia, desia, Sia, DeSia, constructors } = require("sializer");
sia(data)
function serializes the given data using the default parameters.desia(buf)
function deserializes the given buffer using the default parameters.Sia(options)
class makes an instance of Sia serializer using the given options.DeSia(options)
class makes an instance of Sia deserializer using the given options.constructors
is an array of default constructors used both by Sia and DeSia.
The Sia
and DeSia
objects are the core components of the Sia library.
const { sia, desia } = require("sializer");
const buf = sia(data);
const result = desia(buf);
const sia = new Sia({
size = 33554432, // Buffer size to use
constructors = builtinConstructors // An array of extra classes and types
});
const buf = sia.serialize(data);
Where size
is the maximum size of buffer to use, use a big size if you're expecting to
serialize huge objects. The constructors
option is an array of extra types and classes,
it includes instructions for serializing the custom types and classes.
const desia = new DeSia({
mapSize = 256 * 1000, // String map size
constructors = builtinConstructors, // An array of extra classes and types
});
const data = desia.deserialize(buf);
Where mapSize
is the minimum size of string map array to use, use a big size if you're
expecting to serialize huge objects. The constructors
option is an array of extra types
and classes, it includes instructions for deserializing the custom types and classes.
const buf = sia(data);
The sia
function is the Sia.serialize
method on an instance initialized with the default options.
const data = desia(buf);
The desia
function is the DeSia.deserialize
method on an instance initialized with the default options.
The constructors
option is an array of extra types and classes that Sia should support.
Here's an example of how to use it:
const { Sia, DeSia } = require("sializer");
const { constructors: builtins } = require("sializer");
const constructors = [
...builtins,
{
constructor: RegExp, // The custom class you want to support
code: 7, // A unique positive code point for this class, the smaller the better
args: (item) => [item.source, item.flags], // A function to serialize the instances of the class
build(source, flags) { // A function for restoring instances of the class
return new RegExp(source, flags);
},
},
];
const sia = new Sia({ constructors });
const desia = new DeSia({ constructors });
const regex = /[0-9]+/;
const buf = sia.serialize(regex); // serialize the data
const result = desia.deserialize(buf); // deserialize