Control row group size

Question

Control row group size

Closed this issue 2 years ago · 4 comments

Hello, I haven't found how to control row group size.

Yes, I can call Flush, but how do I know if row group reached limit size (1GB for example) ?

Answer 1 · 2022-10-18T16:38:23.000Z

There is currently no control of the row group size in bytes. Since parquet columns are encoded and compressed, I would like to ask what size you would need to control: would it be the compressed size of the row group on disk, or the total decoded size?

Answer 2 · 2022-10-18T17:48:08.000Z

Compressed size of row group

Answer 3 · 2022-10-18T18:45:19.000Z

Do you mind providing a bit more context on the use case that would require controlling the size of a row group on disk?

Answer 4 · 2022-10-25T11:56:00.000Z

We have a big (4GB) parquet file with one row group, and Amazon Athena fails to read with "GENERIC_INTERNAL_ERROR: integer overflow"
We can limit RG by rows number and error disappeared