raveclassic/avsc

Avro for JavaScript :zap:

Avsc

Pure JavaScript implementation of the Avro specification.

Features

• Blazingly fast and compact serialization! Typically faster than JSON with much smaller encodings.
• All the Avro goodness and more: type inference, schema evolution, and remote procedure calls.
• Support for serializing arbitrary JavaScript objects.
• Unopinionated 64-bit integer compatibility.

Installation

$ npm install avsc

avsc is compatible with all versions of node.js since 0.11.

Documentation

• Home
• API
• Quickstart
• Advanced usage
• Benchmarks

Examples

Inside a node.js module, or using browserify:

const avro = require('avsc');

• Encode and decode values from a known schema:

  const type = avro.Type.forSchema({
    type: 'record',
    name: 'Pet',
    fields: [
      {
        name: 'kind',
        type: {type: 'enum', name: 'PetKind', symbols: ['CAT', 'DOG']}
      },
      {name: 'name', type: 'string'}
    ]
  });
  
  const buf = type.toBuffer({kind: 'CAT', name: 'Albert'}); // Encoded buffer.
  const val = type.fromBuffer(buf); // = {kind: 'CAT', name: 'Albert'}

• Infer a value's schema and encode similar values:

  const type = avro.Type.forValue({
    city: 'Cambridge',
    zipCodes: ['02138', '02139'],
    visits: 2
  });
  
  // We can use `type` to encode any values with the same structure:
  const bufs = [
    type.toBuffer({city: 'Seattle', zipCodes: ['98101'], visits: 3}),
    type.toBuffer({city: 'NYC', zipCodes: [], visits: 0})
  ];

• Get a readable stream of decoded values from an Avro container file compressed using Snappy (see the BlockDecoder API for an example including checksum validation):

  const snappy = require('snappy'); // Or your favorite Snappy library.
  const codecs = {
    snappy: function (buf, cb) {
      // Avro appends checksums to compressed blocks, which we skip here.
      return snappy.uncompress(buf.slice(0, buf.length - 4), cb);
    }
  };
  
  avro.createFileDecoder('./values.avro', {codecs})
    .on('metadata', function (type) { /* `type` is the writer's type. */ })
    .on('data', function (val) { /* Do something with the decoded value. */ });

• Implement a TCP server for an IDL-defined protocol:

  // We first generate a protocol from its IDL specification.
  const protocol = avro.readProtocol(`
    protocol LengthService {
      /** Endpoint which returns the length of the input string. */
      int stringLength(string str);
    }
  `);
  
  // We then create a corresponding server, implementing our endpoint.
  const server = avro.Service.forProtocol(protocol)
    .createServer()
    .onStringLength(function (str, cb) { cb(null, str.length); });
  
  // Finally, we use our server to respond to incoming TCP connections!
  require('net').createServer()
    .on('connection', (con) => { server.createChannel(con); })
    .listen(24950);