ankane/ruby-polars

Can't deserialize Decimal column to parquet format

Closed this issue · 3 comments

Hi,

If you try to write a dataframe containing only Decimal column types and then try to read back this parquet file into a polars dataframe, the parsing will fail with the following error:

thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `4`,
 right: `0`', /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow2-0.17.3/src/io/parquet/read/deserialize/primitive/basic.rs:53:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Casting the columns to float fixes the issue although not ideal since some of the decimal columns might actually be integers. Maybe missing this?

Hi @geclos, thanks for reporting. I'm hoping to address once pola-rs/polars#12375 is resolved.

Hi @geclos, with the next release, it'll work if you first call Polars::Config.activate_decimals (added a test case in the commit above).

thanks!