Can't deserialize Decimal column to parquet format
Closed this issue · 3 comments
geclos commented
Hi,
If you try to write a dataframe containing only Decimal column types and then try to read back this parquet file into a polars dataframe, the parsing will fail with the following error:
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
left: `4`,
right: `0`', /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow2-0.17.3/src/io/parquet/read/deserialize/primitive/basic.rs:53:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Casting the columns to float fixes the issue although not ideal since some of the decimal columns might actually be integers. Maybe missing this?
ankane commented
Hi @geclos, thanks for reporting. I'm hoping to address once pola-rs/polars#12375 is resolved.
ankane commented
Hi @geclos, with the next release, it'll work if you first call Polars::Config.activate_decimals
(added a test case in the commit above).
geclos commented
thanks!