Associate Struct (Composite/Complex types) with Record instead of String

[LiuVII]

Currently, dbeam translates Structs as Strings

    "name" : "logo",
    "type" : [ "null", "string" ],
    "doc" : "From sqlType 2002 STRUCT",
    "default" : null,
    "typeName" : "STRUCT",
    "sqlCode" : "2002",
    "columnName" : "logo"

Avro supports nested records/structs and information about type is available in PostgreSQL schema as well, so it seems possible in theory to have this association.

So wonder if it's possible to expand dbeam implementation so in generated schema Structs are mapped to Records?

[labianchin]

I believe it is possible. But it would bring quite some complexity. Some careful design and testing would be needed to map possible combinations of PostgreSQL struct types into Avro types.

For the moment, while DBeam lacks this support, I recommend serializing as json, comma separated or other format via sql views.

[LiuVII]

we went with https://github.com/sksamuel/avro4s and currently building our own data pipelining via PubSub (as we don't have Kafka support atm) and GCS to load data to BQ.

Maybe using avro4s here could be beneficial as well, i.e. to not go deep with translation complexity/testing?
For integration to work with complex types needs custom Encoder/Decoders or Scala case classes to do the trick but like serializing/deserializing to Json via Circe (or smth else) needs about the same, so can be a good match maybe?