Support merge (upsert) in Python
hongbo-miao opened this issue · 12 comments
Description
Use Case
I have dataframe with a few columns timestamp, current, voltage, temperature.
I hope timestamp to be unique, like a primary key.
Currently below code will keep appending if I run same code on same dataframe.
write_deltalake("s3a://my-bucket/my-delta-tables/motor", df, mode="append")It would be great to support merge (upsert) like this https://docs.databricks.com/delta/merge.html
Thanks! 😃
Related Issue(s)
Originally asked at #1355
Hi Team , any updates, of when can this be made available in python
This is blocked on implementing this in Rust, tracked in #850.
Hi Team , any updates, of when can this be made available in python
I have no timeline yet. No one has started working on this yet.
@wjones127 can you assign this to me? Would like to give this a try next few weekends :)
any updates on this? This would be an amazing feature to have out of the box for python. Any open issues blocking this still?
any updates on this? This would be an amazing feature to have out of the box for python. Any open issues blocking this still?
I'm working on it but I expect it will take a couple weeks. I only have time to work on it in weekends
When this will be released to pypi?
When this will be released to pypi?
Somewhere this week
awesome job @ion-elgreco! Thanks for cranking this out!
@ion-elgreco Thanks for enabling this!
Does this feature also enable schema merge? Something similar to this: https://docs.databricks.com/en/delta/update-schema.html#merge-schema-evolution
@ion-elgreco Thanks for enabling this!
Does this feature also enable schema merge? Something similar to this: https://docs.databricks.com/en/delta/update-schema.html#merge-schema-evolution
No only Merge operation.
Is the usage updated in Docs?
@mohitbansal-gep nope not yet. The API has examples for each command and you can find some more clear examples in the blog I wrote: https://delta.io/blog/2023-10-22-delta-rs-python-v0.12.0/