/fastproto

FastProto is a powerful binary data processing tool designed for efficient and flexible binary data serialization and deserialization.

Primary LanguageJavaApache License 2.0Apache-2.0

fastproto

English | 中文

Fast Protocol

Build Status codecov Codacy Badge Maven Central JetBrain Support License

FastProto is a powerful binary data processing tool designed to simplify binary data encoding and decoding in a Java environment. By combining annotations with straightforward APIs, it allows developers to easily manage complex binary data structures, eliminating the need for cumbersome manual coding.

Key Features

  • Annotation-Driven: FastProto utilizes annotations to mark field information in binary data (such as data type, byte offset, endianness, etc.), making parsing and packaging of data fast and intuitive.
  • Extensive Type Support: Supports Java primitive types, unsigned types, string types, time types, arrays, and collection types, catering to diverse data processing needs.
  • Flexible Addressing: Offers reverse addressing capabilities, ideal for handling non-fixed-length binary data, significantly enhancing its application flexibility.
  • Customizable Byte Order: Users can customize the byte order (big-endian or little-endian) to match various data specifications.
  • Encoding and Decoding Formulas: Supports encoding and decoding formulas, including lambda expressions, offering greater flexibility for complex data operations.
  • Diverse APIs: Provides a variety of APIs to suit different application scenarios, ensuring efficiency and reliability in various environments.

Under Developing

  • Code structure & performance optimization
  • Add crc checksum support

Maven

<dependency>
    <groupId>org.indunet</groupId>
    <artifactId>fastproto</artifactId>
    <version>3.10.3</version>
</dependency>

1. Quick Start

Imagine such an application, there is a monitoring device collecting weather data in realtime and sends to the weather station in binary format，the binary data has fixed length of 20 bytes:

65 00 7F 69 3D 84 7A 01 00 00 55 00 F1 FF 0D 00 00 00 07 00

The binary data contains 8 different types of signals, the specific protocol is as follows:

Byte Offset Bit Offset Data Type(C/C++) Signal Name Unit Formula
0 unsigned char device id
1 reserved
2-9 long time ms
10-11 unsigned short humidity %RH
12-13 short temperature ℃
14-17 unsigned int pressure Pa p * 0.1
18 0 bool device valid
18 3-7 reserved
19 reserved

1.1 Decode and encode Binary Data

After the weather station receives the data, it needs to be converted into Java data objects for subsequent business function development. First, define the Java data object Weather according to the protocol, and then use the FastProto data type annotation to annotate each attribute. It should be noted that the offset attribute of annotation corresponds to the byte offset of the signal.

import org.indunet.fastproto.annotation.*;

public class Weather {
    @UInt8Type(offset = 0)
    int id;

    @TimeType(offset = 2)
    Timestamp time;

    @UInt16Type(offset = 10)
    int humidity;

    @Int16Type(offset = 12)
    int temperature;

    @UInt32Type(offset = 14)
    long pressure;

    @BoolType(byteOffset = 18, bitOffset = 0)
    boolean deviceValid;
}

Invoke the FastProto::decode() method to parse the binary data into the Java data object Weather

// datagram sent by monitoring device.
byte[] datagram = ...   

Weather weather = FastProto.decode(datagram, Weather.class);

Invoke the FastProto::encode() method to package the Java data object Weather into binary data. The second parameter of this method is the length of the binary data. If the user does not specify it, FastProto will automatically guess the length.

byte[] datagram = FastProto.encode(weather, 20);

1.2 Transformation Formula

Perhaps you have noticed that the pressure signal corresponds to a conversion formula, usually requiring the user to multiply the serialized result by 0.1, which is an extremely common operation in IoT data exchange. To help users reduce intermediate steps, FastProto introduces decoding formula annotation @DecodingFormula and encoding formula annotation @EncodingFormula, the above simple formula transformation can be implemented by Lambda expression.

import org.indunet.fastproto.annotation.DecodingFormula;
import org.indunet.fastproto.annotation.EncodingFormula;

public class Weather {
    ...

    @UInt32Type(offset = 14)
    @DecodingFormula(lambda = "x -> x * 0.1")
    @EncodingFormula(lambda = "x -> (long) (x * 10)")
    double pressure;
}

2. Annotations

2.1 Primitive Data Type Annotations

FastProto supports Java primitive data types, taking into account cross-language and cross-platform data exchange, unsigned types are also introduced.

Annotation Java C/C++ Size
@BoolType Boolean/boolean bool 1 bit
@AsciiType Character/char char 1 bytes
@CharType Character/char -- 2 bytes
@Int8Type Byte/byte/Integer/int char 1 byte
@Int16Type Short/short / Integer/int short 2 bytes
@Int32Type Integer/int int 4 bytes
@Int64Type Long/long long 8 bytes
@UInt8Type Integer/int unsigned char 1 byte
@UInt16Type Integer/int unsigned short 2 bytes
@UInt32Type Long/long unsigned int 4 bytes
@UInt64Type BigInteger unsigned long 8 bytes
@FloatType Float/float float 4 bytes
@DoubleType Double/double double 8 bytes

2.2 Compound Data Type Annotations

Annotation Java C/C++ Size
@StringType String/StringBuilder/StringBuffer -- N bytes
@TimeType Timestamp/Date/Calendar/Instant long 8 bytes
@EnumType enum enum 1 bytes

2.3 Array Data Type Annotations

Annotation Java C/C++
@BinaryType Byte[]/byte[]/Collection<Byte> char[]
@BoolArrayType Boolean[]/boolean[]/Collection<Boolean> bool[]
@AsciiArrayType Character[]/char[]/Collection<Character> char[]
@CharArrayType Character[]/char[]/Collection<Character> --
@Int8ArrayType Byte[]/byte[]/Integer[]/int[]/Collection<Byte>/Collection<Integer> char[]
@Int16ArrayType Short[]/short[]/Integer[]/int[]/Collection<Short>/Collection<Integer> short[]
@Int32ArrayType Integer[]/int[]/Collection<Integer> int[]
@Int64ArrayType Long[]/long[]/Collection<Long> long[]
@UInt8ArrayType Integer[]/int[]/Collection<Integer> unsigned char[]
@UInt16ArrayType Integer[]/int[]/Collection<Integer> unsigned short[]
@UInt32ArrayType Long[]/long[]/Collection<Long> unsigned int[]
@UInt64ArrayType BigInteger[]/Collection<BigInteger> unsigned long[]
@FloatArrayType Float[]/float[]/Collection<Float> float[]
@DoubleArrayType Double[]/double[]/Collection<Double> double[]

2.4 Supplementary Annotations

FastProto also provides some auxiliary annotations to help users further customize the binary format, decoding and encoding process.

Annotation Scope Description
@DefaultByteOrder Class Default byte order, use little endian if not specified.
@DefaultBitOrder Class Default bit order, use LSB_0 if not specified.
@DecodingIgnore Field Ignore the field when decoding.
@EncodingIgnore Field Ignore the field when encoding.
@DecodingFormula Field Decoding formula.
@EncodingFormula Field Encoding formula.
@AutoType Field Use default type.

2.4.1 Byte Order and Bit Order

FastProto uses little endian by default. You can modify the global byte order through @DefaultByteOrder annotation, or you can modify the byte order of specific field through byteOrder attribute which has a higher priority.

Similarly, FastProto uses LSB_0 by default. You can modify the global bit order through @DefaultBitOrder annotation, or you can modify the bit order of specific field through bitOrder attribute which has a higher priority.

import org.indunet.fastproto.BitOrder;
import org.indunet.fastproto.ByteOrder;
import org.indunet.fastproto.annotation.DefaultBitOrder;
import org.indunet.fastproto.annotation.DefaultByteOrder;

@DefaultByteOrder(ByteOrder.BIG)
@DefaultBitOrder(BitOrder.LSB_0)
public class Weather {
    @UInt16Type(offset = 10, byteOrder = ByteOrder.LITTLE)
    int humidity;

    @BoolType(byteOffset = 18, bitOffset = 0, bitOrder = BitOrder.MSB_0)
    boolean deviceValid;
}

2.4.2 Decoding and Encoding Formula

Users can customize formula in two ways. For simple formulas, it is recommended to use Lambda expression, while for more complex formula, it is recommended to customize formula classes by implementing the java.lang.function.Function interface.

  • Lambda Expression
import org.indunet.fastproto.annotation.DecodingFormula;
import org.indunet.fastproto.annotation.EncodingFormula;

public class Weather {
    ...

    @UInt32Type(offset = 14)
    @DecodingFormula(lambda = "x -> x * 0.1")           // pressure after parsing equals uint32 * 0.1
    @EncodingFormula(lambda = "x -> (long) (x * 10)")   // Data written into binary equals (pressure * 0.1) cast to long
    double pressure;
}
  • Custom Formula Class
import java.util.function.Function;

public class PressureDecodeFormula implements Function<Long, Double> {
    @Override
    public Double apply(Long value) {
        return value * 0.1;
    }
}
import java.util.function.Function;

public class PressureEncodeFormula implements Function<Double, Long> {
    @Override
    public Long apply(Double value) {
        return (long) (value * 10);
    }
}
import org.indunet.fastproto.annotation.DecodingFormula;
import org.indunet.fastproto.annotation.EncodingFormula;

public class Weather {
    ...

    @UInt32Type(offset = 14)
    @DecodingFormula(PressureDecodeFormula.class)
    @EncodingFormula(PressureEncodeFormula.class)
    double pressure;
}

Users can specify only the encoding formula or only the decoding formula as needed. If both lambda expression and custom formula class are specified, the latter has a higher priority.

2.4.3 AutoType

FastProto can automatically infer type if field is annotated by @AutoType.

import org.indunet.fastproto.annotation.AutoType;

public class Weather {
    @AutoType(offset = 10, byteOrder = ByteOrder.LITTLE)
    int humidity;   // default Int32Type

    @AutoType(offset = 14)
    long pressure;  // default Int64Type
}

2.4.4 Ignore Field

In special cases, if you want to ignore certain fields during parsing, or ignore certain fields during packaging, you can use @DecodingIgnore and @EncodingIgnore.

import org.indunet.fastproto.annotation.*;

public class Weather {
    @DecodingFormula
    @Int16Type(offset = 10)
    int humidity;   // ignore when parsing

    @EncodingIgnore
    @Int32Type(offset = 14)
    long pressure; // ignore when packaging
}

3. Scala

FastProto supports case class，but Scala is not fully compatible with Java annotations, so please refer to FastProto as follows.

import org.indunet.fastproto.annotation.scala._

4. Decode and encode without Annotations

In some special cases, developers do not want or cannot use annotations to decorate data objects, for example, data objects come from third-party libraries, developers cannot modify the source code, and developers only want to create binary data blocks in a simple way. FastProto provides simple API to solve the above problems, as follows:

4.1 Decode Binary Data

  • Decode with data object
byte[] bytes = ... // Binary data to be decoded

public class DataObject {
    Boolean f1;
    Integer f2;
    Integer f3;
}

DataObject obj = FastProto.decode(bytes)
        .readBool("f1", 0, 0)       // Decode boolean data at byte offset 0 and bit offset 0
        .readInt8("f2", 1)          // Decode signed 8-bit integer data at byte offset 1
        .readInt16("f3", 2)         // Decode signed 8-bit integer data at byte offset 2
        .mapTo(DataObject.class);   // Map decoded result into Java data object according to the field name
  • Decode without data object
import org.indunet.fastproto.util.DecodeUtils;

byte[] bytes = ... // Binary data to be decoded
        
boolean f1 = DecodeUtils.readBool(bytes, 0, 0); // Decode boolean data at byte offset 0 and bit offset 0
int f2 = DecodeUtils.readInt8(bytes, 1);        // Decode signed 8-bit integer data at byte offset 1
int f3 = DecodeUtils.readInt16(bytes, 2);       // Decode signed 8-bit integer data at byte offset 2

4.2 Create Binary Data Block

byte[] bytes = FastProto.create(16)         // Create binary block with 16 bytes 
        .writeInt8(0, 1)                    // Write unsigned 8-bit integer 1 at byte offset 0
        .writeUInt16(2, 3, 4)               // Write 2 unsigned 16-bit integer 3 and 4 consecutively at byte offset 2
        .writeUInt32(6, ByteOrder.BIG, 256)  // Write unsigned 32-bit integer 256 at byte offset 6 with big endian
        .get();
import org.indunet.fastproto.util.EncodeUtils;

byte[] bytes = new byte[16];

EncodeUtils.writeInt8(bytes, 0, 1);                     // Write unsigned 8-bit integer 1 at byte offset 0
EncodeUtils.writeUInt16(bytes, 2, 3, 4);                // Write 2 unsigned 16-bit integer 3 and 4 consecutively at byte offset 2
EncodeUtils.writeUInt32(bytes, 6, ByteOrder.BIG, 256);  // Write unsigned 32-bit integer 256 at byte offset 6 with big endian

5. Benchmark

  • windows 11, i7 11th, 32gb
  • openjdk 1.8.0_292
  • binary data of 60 bytes and protocol class of 13 fields
  1. api with annotations
Benchmark Mode Samples Score Error Units
FastProto::decode throughput 10 240 ± 4.6 ops/ms
FastProto::encode throughput 10 317 ± 11.9 ops/ms
  1. api without annotations
Benchmark Mode Samples Score Error Units
FastProto::decode throughput 10 1273 ± 17 ops/ms
FastProto::create throughput 10 6911 ± 162 ops/ms

6. Build Requirements

  • Java 1.8+
  • Maven 3.5+

7. Contribution

FastProto has obtained the support of JetBrain Open Source Project, which can provide free license of all product pack for all core contributors. If you are interested in this project and want to join and undertake part of the work (development/testing/documentation), please feel free to contact me via email deng_ran@foxmail.com

8. License

FastProto is released under the Apache 2.0 license.

Copyright 2019-2021 indunet.org

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at the following link.

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.