Feature request: Progress bar for full table copies
Closed this issue · 6 comments
I use odbc2parquet to make full copies of tables from SQL Server.
Would be nice to have odbc2parquet show a progress bar if it knows how many rows the query result will have. For full table copies this is easy to achieve using count_big(*).
At the very least it would be nice for odbc2parquet to log the total number of rows fetched so far, not only the current batch number and size.
Currently it is entirely possible to use odbc2parquet in situations there the first files has been written while even the data source is not sure how many rows the query will yield. odbc2parquet has no idea what the maximum numbers of row are. I am not confident that every data source would support count_big or that it would always be cheap to call.
Emitting the total number of rows fetched so far on the other hand is easy. I see nothing that would speak against that.
Yeah, the count thing would only make sense with a special "full table copy" mode, where you're not passing a query but a table name.
odbc2parquet 3.1.0 is released which emits a log message on info level stating the number of currently fetched rows.
I have thought a bit about how to do a progress bar, but there is no good way to retrieve the total number of rows without making additional assumptions about the data source in question. The age old tradeoff between usefulness and re-usability.
However I could consider making the core functions of this CLI tool available as the library, allowing you to build a specialized CLI tool for your use-case on top. I would only extend the mental effort if this is something you would consider, though. Otherwise I am also fine with closing the issue and calling it a day.
Best, Markus
Ok! I don't think I would use that library given that I'm much more fluent in Python and there are various easy to use ways to fetch data from SQL Server and write to Parquet in Python.
Thanks for the feedback, closing the issue.