PY To Proto

This library holds utilities for converting in-memory data schema representations to Protobuf. The intent is to allow python libraries to leverage the power of protobuf while maintaining the source-of-truth for their data in pure python and avoiding static build steps.

Why?

The protobuf langauge is a powerful tool for defining language-agnostic, composable datastructures. Protobuf also offers cross-language compatibility so that a given set of definitions can be compiled into numerous target programming languages. The downside is that protobuf requires_a static built step to perform this proto -> X conversion step. Alternately, there are multiple ways of representing data schemas in pure python which allow a python library to interact with well-typed data objects. The downside here is that these structures can not easily be used from other programming languages. The pros/cons of these generally fall along the following lines:

Protobuf:
- Advantages
  - Compact serialization
  - Auto-generated grpc client and service libraries
  - Client libraries can be used from different programming languages
- Disadvantages
  - Learning curve to understand the full ecosystem
  - Not a familiar tool outside of service engineering
  - Static compilation step required to use in code
Python schemas:
- Advantages
  - Can be learned quickly using pure-python documentation
  - Can be written inline in pure python
- Disadvantages
  - Generally, no standard serialization beyond json
  - No automated service implementations
  - No/manual mechanism for usage in other programming languages

This project aims to bring the advantages of both types of schema representation so that a given project can take advantage of the best of both:

Define your structures in pure python for simplicity
Dynamically create google.protobuf.Descriptor objects to allow for protobuf serialization and deserialization
Reverse render a .proto file from the generated Descriptor so that stubs can be generated in other languages
No static compiliation needed!

Supported Python Schema Types

Currently, objects can be declared using either python dataclasses or Json TypeDef (JTD). Additional schemas can be added by subclassing ConverterBase.

Dataclass To Proto

The following example illustrates how dataclasses and enums can be converted to proto:

from dataclasses import dataclass
from enum import Enum
from typing import Annotated, Dict, List, Enum
import py_to_proto

# Define the Foo structure as a python dataclass, including a nested enum
@dataclass
class Foo:

    class BarEnum(Enum):
        EXAM: 0
        JOKE_SETTING: 1

    foo: bool
    bar: List[BarEnum]

# Define the Foo protobuf message class
FooProto = py_to_proto.descriptor_to_message_class(
    py_to_proto.dataclass_to_proto(
        package="foobar",
        dataclass_=Foo,
    )
)

# Declare the Bar structure as a python dataclass with a reference to the
# FooProto type
@dataclass
class Bar:
    baz: FooProto

# Define the Bar protobuf message class
BarProto = py_to_proto.descriptor_to_message_class(
    py_to_proto.dataclass_to_proto(
        package="foobar",
        dataclass_=Bar,
    )
)

# Instantiate a BarProto
print(BarProto(baz=FooProto(foo=True, bar=[Foo.BarEnum.EXAM.value])))

def write_protos(proto_dir: str):
    """Write out the .proto files for FooProto and BarProto to the given
    directory
    """
    FooProto.write_proto_file(proto_dir)
    BarProto.write_proto_file(proto_dir)

JTD To Proto

The following example illustrates how JTD schemas can be converted to proto:

import py_to_proto

# Declare the Foo protobuf message class
Foo = py_to_proto.descriptor_to_message_class(
    py_to_proto.jtd_to_proto(
        name="Foo",
        package="foobar",
        jtd_def={
            "properties": {
                # Bool field
                "foo": {
                    "type": "boolean",
                },
                # Array of nested enum values
                "bar": {
                    "elements": {
                        "enum": ["EXAM", "JOKE_SETTING"],
                    }
                }
            }
        },
    )
)

# Declare an object that references Foo as the type for a field
Bar = py_to_proto.descriptor_to_message_class(
    py_to_proto.jtd_to_proto(
        name="Bar",
        package="foobar",
        jtd_def={
            "properties": {
                "baz": {
                    "type": Foo.DESCRIPTOR,
                },
            },
        },
    ),
)

def write_protos(proto_dir: str):
    """Write out the .proto files for Foo and Bar to the given directory"""
    Foo.write_proto_file(proto_dir)
    Bar.write_proto_file(proto_dir)

Similar Projects

There are a number of similar projects in this space that offer slightly different value: