duckdb/duckdb_iceberg

Support AWS Glue Catalog

rustyconover opened this issue · 4 comments

rustyconover commented

I'd like to add support for "schemes" that prefix the location supplied for Iceberg tables.

For example I'd like to support AWS Glue with a scheme like:

iceberg_scan('aws-glue://[catalog id].[database_name].[table_name]').

The use of this scheme would allow DuckDB to access Iceberg tables where the current snapshot version is stored in a Glue data catalog. The Iceberg extension would call Glue's describe_table() API method, get the current metadata_url and parse the JSON file.

I'd also like to add a companion function called iceberg_scan_tables('aws-glue://[catalog id]') that would scan all databases and tables under the schema and create views that are backed by iceberg_scan(), so this can expose the databases and tables in glue to the end user.

a-agmon commented

Might be relevant here too :
#22

pixie79 commented

Would be great to see glue supported here as an option - especially with Lake Formation as well

rustyconover commented

I have a pending PR for this functionality.

  • 🎉1
rustyconover commented

#51 should close this issue.

  • 🚀1