TinyBin Protocol Specification

TinyBin is a high-performance binary serialization library for Go, specifically designed for TinyGo compatibility. It's an adaptation of the Kelindar/binary project, optimized for resource-constrained environments and embedded systems.

Features

TinyGo Compatible: Designed to work seamlessly with TinyGo compiler
High Performance: Efficient binary encoding/decoding with minimal allocations
Type Safe: Compile-time type checking with reflection-based runtime support
Extensible: Custom codec support through the Codec interface
Memory Efficient: Object pooling for encoders and decoders

Installation

go get github.com/cdvelop/tinybin

Quick Start

package main

import (
    "fmt"
    "github.com/cdvelop/tinybin"
)

// Define a struct to serialize
type User struct {
    Name    string  `json:"name"`
    Age     int     `json:"age"`
    Active  bool    `json:"active"`
    Balance float64 `json:"balance"`
}

func main() {
    // Create an instance
    user := &User{
        Name:    "Alice",
        Age:     30,
        Active:  true,
        Balance: 123.45,
    }

    // Create TinyBin instance
    tb := tinybin.New()

    // Encode to binary
    data, err := tb.Encode(user)
    if err != nil {
        panic(err)
    }

    // Decode from binary
    var decoded User
    err = tb.Decode(data, &decoded)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Decoded: %+v\n", decoded)
}

Core API

Encoding Functions

`(*TinyBin) Encode(v any) ([]byte, error)`

Encodes any value into binary format and returns the resulting bytes.

tb := tinybin.New()
data, err := tb.Encode(myStruct)

`(*TinyBin) EncodeTo(v any, dst io.Writer) error`

Encodes a value directly to an io.Writer.

tb := tinybin.New()
var buf bytes.Buffer
err := tb.EncodeTo(myStruct, &buf)

Decoding Functions

`(*TinyBin) Decode(b []byte, v any) error`

Decodes binary data into a value. The destination must be a pointer.

tb := tinybin.New()
var result MyStruct
err := tb.Decode(data, &result)

encoder Type

Note: Encoders are now managed internally by TinyBin instances through object pooling for better performance and resource management. Direct creation of encoders is deprecated.

`(*encoder) Encode(v any) error`

Encodes a value using the encoder instance.

tb := tinybin.New()
var buffer bytes.Buffer
err := tb.EncodeTo(myValue, &buffer) // Uses pooled encoder internally

`(*encoder) Buffer() io.Writer`

Returns the underlying writer.

tb := tinybin.New()
var buffer bytes.Buffer
tb.EncodeTo(myValue, &buffer)
writer := buffer // Direct access to buffer

encoder Write Methods

The encoder type provides methods for writing primitive types:

Write(p []byte) - writes raw bytes
WriteVarint(v int64) - writes a variable-length signed integer
WriteUvarint(x uint64) - writes a variable-length unsigned integer
WriteUint16(v uint16) - writes a 16-bit unsigned integer
WriteUint32(v uint32) - writes a 32-bit unsigned integer
WriteUint64(v uint64) - writes a 64-bit unsigned integer
WriteFloat32(v float32) - writes a 32-bit floating point number
WriteFloat64(v float64) - writes a 64-bit floating point number
WriteBool(v bool) - writes a boolean value
WriteString(v string) - writes a string with length prefix

decoder Type

Note: Decoders are now managed internally by TinyBin instances through object pooling for better performance and resource management. Direct creation of decoders is deprecated.

`(*TinyBin) Decode(data []byte, v any) error`

Decodes binary data into a value using the TinyBin instance. The destination must be a pointer.

tb := tinybin.New()
var result MyStruct
err := tb.Decode(data, &result)

decoder Read Methods

The decoder type provides methods for reading primitive types:

Read(b []byte) (int, error) - reads raw bytes
ReadVarint() (int64, error) - reads a variable-length signed integer
ReadUvarint() (uint64, error) - reads a variable-length unsigned integer
ReadUint16() (uint16, error) - reads a 16-bit unsigned integer
ReadUint32() (uint32, error) - reads a 32-bit unsigned integer
ReadUint64() (uint64, error) - reads a 64-bit unsigned integer
ReadFloat32() (float32, error) - reads a 32-bit floating point number
ReadFloat64() (float64, error) - reads a 64-bit floating point number
ReadBool() (bool, error) - reads a boolean value
ReadString() (string, error) - reads a length-prefixed string
Slice(n int) ([]byte, error) - returns a slice of the next n bytes
ReadSlice() ([]byte, error) - reads a variable-length byte slice

TinyBin Constructor and Instance Architecture

Creating Instances

`New(args ...any) *TinyBin`

Creates a new TinyBin instance with optional configuration. Each instance is completely isolated from others.

// Basic instance (no logging)
tb := tinybin.New()

// With custom logging
tb := tinybin.New(func(msg ...any) {
    log.Printf("TinyBin: %v", msg)
})

Instance Isolation Benefits

Complete State Isolation: Each TinyBin instance maintains its own:

Schema cache (slice-based for TinyGo compatibility)
encoder and decoder object pools
Optional logging function

Thread Safety: Multiple goroutines can safely use the same instance concurrently without external synchronization.

Testing Benefits: Each test can create its own instance with custom logging for complete isolation.

func TestMyFunction(t *testing.T) {
    // Completely isolated test instance
    tb := tinybin.New(func(msg ...any) {
        t.Logf("TinyBin: %v", msg)
    })

    data, err := tb.Encode(testData)
    assert.NoError(t, err)
}

Multiple Instance Patterns

Microservices Pattern: Different services can use separate instances for complete isolation.

type ProtocolManager struct {
    httpTinyBin  *tinybin.TinyBin
    grpcTinyBin  *tinybin.TinyBin
    kafkaTinyBin *tinybin.TinyBin
}

func NewProtocolManager() *ProtocolManager {
    return &ProtocolManager{
        httpTinyBin:  tinybin.New(), // Production: no logging
        grpcTinyBin:  tinybin.New(),
        kafkaTinyBin: tinybin.New(),
    }
}

Concurrent Processing: Multiple instances can be used safely across goroutines.

// Each goroutine gets its own instance for complete isolation
go func() {
    tb := tinybin.New()
    data, _ := tb.Encode(data1)
    process(data)
}()

go func() {
    tb := tinybin.New()
    data, _ := tb.Encode(data2) // Completely independent
    process(data)
}()

Supported Data Types

TinyBin automatically handles encoding and decoding for the following types:

Primitive Types

bool - encoded as a single byte (0 or 1)
int, int8, int16, int32, int64 - variable-length encoded
uint, uint8, uint16, uint32, uint64 - variable-length encoded
float32, float64 - IEEE 754 binary representation
string - UTF-8 bytes with length prefix

Composite Types

Slices - length-prefixed sequence of elements

[]int{1, 2, 3, 4, 5}     // → [5, 1, 2, 3, 4, 5]
[]string{"a", "b", "c"}   // → [3, "a", "b", "c"]

Arrays - fixed-size sequence of elements

[3]int{1, 2, 3}          // → [1, 2, 3]

Structs - field-by-field encoding
```
type Point struct {
    X, Y int
}
```

Pointers - nil check followed by element encoding

var ptr *MyStruct = &MyStruct{...}  // → [0, ...data...]
var nilPtr *MyStruct = nil          // → [1]

Codec Interface

For custom types, implement the Codec interface:

type Codec interface {
    EncodeTo(*encoder, reflect.Value) error
    DecodeTo(*decoder, reflect.Value) error
}

Utility Functions

`ToString(b *[]byte) string`

Converts a byte slice to string without allocation (unsafe operation).

`ToBytes(v string) []byte`

Converts a string to byte slice without allocation (unsafe operation).

Advanced Usage

Multiple Instance Usage

// Create multiple isolated instances
httpTB := tinybin.New()
grpcTB := tinybin.New()
kafkaTB := tinybin.New()

// Each instance maintains its own cache and pools
httpData, _ := httpTB.Encode(data)
grpcData, _ := grpcTB.Encode(data)
kafkaData, _ := kafkaTB.Encode(data)

Custom Instance with Logging

// Create instance with custom logging for debugging
tb := tinybin.New(func(msg ...any) {
    log.Printf("TinyBin Debug: %v", msg)
})

// Use like normal
data, err := tb.Encode(myStruct)
if err != nil {
    log.Printf("Encoding failed: %v", err)
}

Concurrent Usage

tb := tinybin.New()

// Safe concurrent usage - internal pooling handles synchronization
go func() {
    data, _ := tb.Encode(data1)
    process(data)
}()

go func() {
    data, _ := tb.Encode(data2)
    process(data)
}()

Error Handling

tb := tinybin.New()

data, err := tb.Encode(myValue)
if err != nil {
    // Handle encoding error
    log.Printf("Encoding failed: %v", err)
}

var result MyType
err = tb.Decode(data, &result)
if err != nil {
    // Handle decoding error
    log.Printf("Decoding failed: %v", err)
}

Migration from Global API

Quick Migration

If you're upgrading from the previous global function API, here's how to migrate:

Before (Global Functions)

// Old global API
data, err := tinybin.Encode(myStruct)
err = tinybin.Decode(data, &result)

After (Instance API)

// New instance API
tb := tinybin.New()
data, err := tb.Encode(myStruct)
err = tb.Decode(data, &result)

Benefits of Migration

Complete Isolation: No shared state between different parts of your application
Better Testing: Each test can have its own isolated instance
Thread Safety: Multiple instances can be used safely across goroutines
TinyGo Compatible: Slice-based caching instead of sync.Map for embedded targets

Common Migration Patterns

Simple Replacement

// Replace all instances of:
tinybin.Encode(data)
tinybin.Decode(data, &result)
tinybin.EncodeTo(data, &buffer)

// With:
tb := tinybin.New()
tb.Encode(data)
tb.Decode(data, &result)
tb.EncodeTo(data, &buffer)

Service Integration

type MyService struct {
    tb *tinybin.TinyBin
}

func NewMyService() *MyService {
    return &MyService{
        tb: tinybin.New(), // Instance per service
    }
}

Testing Migration

func TestMyFunction(t *testing.T) {
    // Old way: Global state could interfere
    // data, _ := tinybin.Encode(testData)

    // New way: Completely isolated
    tb := tinybin.New()
    data, err := tb.Encode(testData)
    assert.NoError(t, err)
}

Performance Considerations

Instance-Based Pooling: Each TinyBin instance maintains its own encoder and decoder pools for optimal resource management
Zero Allocations: Where possible, operations avoid heap allocations for maximum performance
Variable-Length Integers: Integers are encoded with minimal bytes using efficient algorithms
Unsafe Operations: String/byte conversions use unsafe operations for performance when appropriate
Slice-Based Caching: TinyGo-compatible slice-based schema cache provides fast lookups with minimal memory overhead
Complete Isolation: Multiple instances can operate concurrently without contention, improving scalability in multi-goroutine environments

Dependencies

github.com/cdvelop/tinystring - String utilities

License

This project is an adaptation of https://github.com/Kelindar/binary focused on TinyGo compilation targets.

cdvelop/tinybin