mmolimar/kafka-connect-fs

Parquet dependency should be updated to 1.12+ to support the INT96 compatibility flag

rjbarrington opened this issue · 0 comments

The current parquet dependency is for the 1.11.x version, which doesn't support INT96 types. Older datasets generated by Spark applications may have used this type. Parquet 1.12+ has added support for this. Updating to 1.12.3 allows the user to add "policy.fs.parquet.avro.readInt96AsFixed": "true", for example.

See https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration, https://issues.apache.org/jira/browse/PARQUET-1928, https://github.com/apache/parquet-mr/blob/master/CHANGES.md.