Skip to content

datatorch/avsc

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Avsc NPM version Download count Build status Coverage status

Pure JavaScript implementation of the Avro specification.

Features

Installation

$ npm install avsc

avsc is compatible with all versions of node.js since 0.11 and major browsers via browserify. For convenience, you can also find compiled distributions with the releases (but please host your own copy).

Documentation

Examples

Inside a node.js module, or using browserify:

const avro = require('avsc');
  • Encode and decode values from a known schema:

    const type = avro.Type.forSchema({
      type: 'record',
      fields: [
        {name: 'kind', type: {type: 'enum', symbols: ['CAT', 'DOG']}},
        {name: 'name', type: 'string'}
      ]
    });
    
    const buf = type.toBuffer({kind: 'CAT', name: 'Albert'}); // Encoded buffer.
    const val = type.fromBuffer(buf); // = {kind: 'CAT', name: 'Albert'}
  • Infer a value's schema and encode similar values:

    const type = avro.Type.forValue({
      city: 'Cambridge',
      zipCodes: ['02138', '02139'],
      visits: 2
    });
    
    // We can use `type` to encode any values with the same structure:
    const bufs = [
      type.toBuffer({city: 'Seattle', zipCodes: ['98101'], visits: 3}),
      type.toBuffer({city: 'NYC', zipCodes: [], visits: 0})
    ];
  • Get a readable stream of decoded values from an Avro container file compressed using Snappy (see the BlockDecoder API for an example including checksum validation):

    const snappy = require('snappy'); // Or your favorite Snappy library.
    const codecs = {
      snappy: function (buf, cb) {
        // Avro appends checksums to compressed blocks, which we skip here.
        return snappy.uncompress(buf.slice(0, buf.length - 4), cb);
      }
    };
    
    avro.createFileDecoder('./values.avro', {codecs})
      .on('metadata', function (type) { /* `type` is the writer's type. */ })
      .on('data', function (val) { /* Do something with the decoded value. */ });
  • Implement a TCP server for an IDL-defined protocol:

    // We first generate a protocol from its IDL specification.
    const protocol = avro.readProtocol(`
      protocol LengthService {
        /** Endpoint which returns the length of the input string. */
        int stringLength(string str);
      }
    `);
    
    // We then create a corresponding server, implementing our endpoint.
    const server = avro.Service.forProtocol(protocol)
      .createServer()
      .onStringLength(function (str, cb) { cb(null, str.length); });
    
    // Finally, we use our server to respond to incoming TCP connections!
    require('net').createServer()
      .on('connection', (con) => { server.createChannel(con); })
      .listen(24950);

Packages

No packages published

Languages

  • JavaScript 96.8%
  • Python 1.7%
  • Java 1.2%
  • Other 0.3%