file-size-threshold generates wrong size files
Rickitik opened this issue · 1 comments
Rickitik commented
i tried to use odbc2parquet with following options :
odbc2parquet query --batch-size-memory 100Mb --column-compression-default zstd --file-size-threshold 1GiB --no-empty-file --connection-string "some connection string" "/code/data/processed/test_file_from_server.par" "some sql query"
If i use this query without file-size limitation i get 1.8Gb result file.
If i use limitation, first file is generated in right way ~ 1Gb, i expected to gain next one with size near 800 Mb, but i got lots of files with 10-15kb size. (total sum - near 800 Mb). 10-15Kb - the size of one row-group