Write functionality
adriangb opened this issue · 4 comments
This is a “far out there” feature request that I suspect is not even possible to implement (it’s a scanner not a writer after all), but here it goes anyway. Any chance DuckDB could grow Postgres writing functionality?
We have data pipelines where we move data from an analytical database (Redshift and BigQuery) to Postgres. We currently do this by dumping CSVs which we then load into Postgres. This is a pain most of the time and a nightmare when you start running into delimiters in your text columns, nulls vs empty characters, etc. It would be really nice to be able to go from Parquet to Postgres (or even OLAP DB to Postgres) using the simplicity of DuckDB’s ability to scan, buffer and transform data.
Just a thought after a night of hair pulling 😅
Thank you for your thoughts! @Mytherin already has some magic up his sleeves in this direction, but it isn't something we have shared just yet.
Stay tuned!!
In the meantime though, would the DuckDB Foreign Data Wrapper work for you?
https://github.com/alitrack/duckdb_fdw
Cool, looking forward to it!
Unfortunately Postgres extensions aren’t really an option in my case since CloudSQL (which is the Postgres we’re using) doesn’t let you install extensions beyond those already bundled. But that’s a great idea that I’ll keep in mind if I run into this with vanilla Postgres!
+1 to this idea. Ran into some perfect use cases for it a couple days ago.