Allow the use of Catalog Providers on sc2 upserts.
PadenZach opened this issue · 3 comments
👋 Hello all!
Looking into integrating mack with some of our internal tooling, and one thing I noticed was that the sc2 upsert functions are written in a way such that they only support paths.
Ideally, I'd like these to work for Catalog providers as well, for example:
- Hive Catalogs
- Glue Catalogs
- Unity Catalog / Databricks
Is this a desired feature, or is using catalog not best-practice (Please let me know if it's not, and I'll pivot some internally tooling 😅)
If it's just due to the greenness of the mack project, I'd be willing to work on a PR.
Thanks!
@PadenZach - thanks for reporting this. Yea, we should definitely be writing all functions such that they work for paths and names. We should never be hardcoding DeltaTable.forPath in a function - so happy you're reporting this. We should update this method to accept the Delta table as an argument, not the path. Would you be willing to submit a pull request for this?
Yeah I can work on putting something together.
Would you be more interested in a PR refactoring everything to take a delta table, or a PR that refactors them to take table, fs path, or catalog name? Not sure how concerned with API changes the project is at this point.
@PadenZach - the goal is to make mack work in every environment and also keep it dependency free. Would we need to add any catalog logic? Wouldn't it be possible to keep the catalog logic outside of Mack?