pacman82/odbc2parquet

export in chunks?

Closed this issue · 3 comments

sashk8 commented

Hi!

First of all - thank you for this great tool. :)

Is there a way to make the export in chunks?

I would like to define maximum amount of rows per file - so that the exported data is written into multiple files.

pacman82 commented

Hello @sashk8 ,

First of all - thank you for this great tool. :)

Thank you

Is there a way to make the export in chunks?

There are two ways even, to limit the size of each individual file:

--row-groups-per-file and --file-size-threshold. odbc2parquet query --help will tell you more.

I would be happy to learn if these can solve your problem.

Cheers, Markus

sashk8 commented

Hi @pacman82,

Thank you. This solves my problem completely. :) I didn't know that all these parameters exist.

Great tool. I was testing several ways how to export data from MSSQL to Parquet and this tool is by far the fastest. 👍

Best regards,
Sašo

pacman82 commented

Happy to hear that