spotify/magnolify

Support Avro logical types

nevillelyh opened this issue · 2 comments

Specifically the ones used in BigQuery Avro support:
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

We might need more than just the field type T to be able to do this though, like need to check record schema to verify logicalType field.

Doc of Avro mapping for BigQuery Storage API:
https://cloud.google.com/bigquery/docs/reference/storage

And actual schema from a test table:

{
  "type" : "record",
  "name" : "__root__",
  "fields" : [ {
    "name" : "date",
    "type" : {
      "type" : "int",
      "logicalType" : "date"
    }
  }, {
    "name" : "time",
    "type" : {
      "type" : "long",
      "logicalType" : "time-micros"
    }
  }, {
    "name" : "datetime",
    "type" : {
      "type" : "string",
      "logicalType" : "datetime"
    }
  }, {
    "name" : "timestamp",
    "type" : {
      "type" : "long",
      "logicalType" : "timestamp-micros"
    }
  }, {
    "name" : "numeric",
    "type" : {
      "type" : "bytes",
      "logicalType" : "decimal",
      "precision" : 38,
      "scale" : 9
    }
  }, {
    "name" : "geography",
    "type" : {
      "type" : "string",
      "sqlType" : "GEOGRAPHY"
    }
  } ]
}

Also re: NUMERIC rounding: https://cloud.google.com/bigquery/docs/reference/standard-sql/conversion_rules

Fixed in 08c3ad3