Parquet dependency should be updated to 1.12+ to support the INT96 compatibility flag
rjbarrington opened this issue · 0 comments
rjbarrington commented
The current parquet dependency is for the 1.11.x version, which doesn't support INT96 types. Older datasets generated by Spark applications may have used this type. Parquet 1.12+ has added support for this. Updating to 1.12.3 allows the user to add "policy.fs.parquet.avro.readInt96AsFixed": "true",
for example.
See https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration, https://issues.apache.org/jira/browse/PARQUET-1928, https://github.com/apache/parquet-mr/blob/master/CHANGES.md.