token-types

A primitive token library used to read and write from a Uint8Array. Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.

Compatibility

Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM). JavaScript is compliant with ECMAScript 2019 (ES10).

Installation

npm install --save token-types

Usually in combination with strtok3:

npm install --save strtok3

Using TypeScript you should also install @tokenizer/token as a development dependency:

npm install --save-dev @tokenizer/token

Example

import * as strtok3 from 'strtok3';
import * as token from 'token-types';

(async () => {

  const tokenizer = await strtok3.fromFile("somefile.bin");
  try {
    const myNumber = await tokenizer.readToken(token.Float32_BE);
    console.log(`My number: ${myNumber}`);
  } finally {
    tokenizer.close(); // Close the file
  }
})();

Tokens

Numeric tokens

token-types supports a wide variety of numeric tokens out of the box:

Token	Number	Bits	Endianness
`UINT8`	Unsigned integer	8	n/a
`UINT16_BE`	Unsigned integer	16	big endian
`UINT16_LE`	Unsigned integer	16	little endian
`UINT24_BE`	Unsigned integer	24	big endian
`UINT24_LE`	Unsigned integer	24	little endian
`UINT32_BE`	Unsigned integer	32	big endian
`UINT32_LE`	Unsigned integer	32	little endian
`UINT64_BE`	Unsigned integer	64	big endian
`UINT64_LE`*	Unsigned integer	64	little endian
`INT8`	Signed integer	8	n/a
`INT16_BE`	Signed integer	16	big endian
`INT16_LE`	Signed integer	16	little endian
`INT24_BE`	Signed integer	24	big endian
`INT24_LE`	Signed integer	24	little endian
`INT32_BE`	Signed integer	32	big endian
`INT32_LE`	Signed integer	32	little endian
`INT64_BE`	Signed integer	64	big endian
`INT64_LE`*	Signed integer	64	little endian
`Float16_BE`	IEEE 754 float	16	big endian
`Float16_LE`	IEEE 754 float	16	little endian
`Float32_BE`	IEEE 754 float	32	big endian
`Float32_LE`	IEEE 754 float	32	little endian
`Float64_BE`	IEEE 754 float	64	big endian
`Float64_LE`	IEEE 754 float	64	little endian
`Float80_BE`*	IEEE 754 float	80	big endian
`Float80_LE`*	IEEE 754 float	80	little endian

(*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

String tokens

StringType decoding is implemented using TextDecoder which supports a large number of encodings including but not limited to:

UTF-8 (the default)
Windows-1252
ISO-8859-1

Check out the MDN web docs for the TextDecoder for a complete list

Custom tokens

Custom tokens can be added, suitable for reading binary files or network messages:

  ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
      };
    }
  };

Borewit/token-types