Dunky

A Jupyter Kernel for DuckDB with Unity Catalog.

Description

Dunky is a Jupyter kernel that allows you to run DuckDB queries with Unity Catalog integration directly from your Jupyter notebooks.

I created this extension because existing solutions such as jupysql require you to use magics, load uc_catalog, delta, and manage secrets and don't work well with duckdb's uc_catalog extension.

Features

Run DuckDB queries in Jupyter notebooks
Unity Catalog integration
No need to use magics
Nice output formatting
No need to load uc_catalog, delta and manage secrets
CREATE EXTERNAL TABLE [table_name] LOCATION [location] OPTIONS [options] to create a Unity Catalog delta table

Installation

To install Dunky, you can use the following commands:

pip install dunky

Configure Unity Catalog

You can set the following environment variables to configure Unity Catalog:

UC_ENDPOINT: The endpoint of the Unity Catalog server.
UC_TOKEN: The token to authenticate with the Unity Catalog server.
UC_AWS_REGION: The AWS region to use for the Unity Catalog server.

These settings default to localhost:8080/api/2.1/unity-catalog, not-used, and eu-west-1 respectively.

Usage

After installing, you can start using the Dunky kernel in your Jupyter notebooks. Select the "Dunky" kernel from the kernel selection menu.

You can directly query DuckDB tables and use Unity Catalog features in your notebooks. You don't need to set up a connection or manage credentials, as Dunky handles all of that for you.

Start with attaching your database using the ATTACH DATABASE command. e.g.,

ATTACH DATABASE 'unity' AS unity (TYPE UC_CATALOG);

After attaching, just start writing your queries and enjoy the power of DuckDB with Unity Catalog integration!

S3 Integration

Dunky supports AWS S3 integration with Unity Catalog.

prerequisite:
- Make sure the unity catalog has S3 bucket authentication configured
Writing to S3: in the CREATE EXTERNAL TABLE set location to s3://your-bucket-name

ps. Dunky might also work with gcp and azure, but have not tested this. depends on whether unity and duckdb uc_catalog support it. I've seen some people confirming that unity catalog and duckdb can work with Azure and gcp.

Example docker

In the docker folder, you can find an example of how to run JupyterLab with Dunky and Unity Catalog in Docker containers. To run the example, execute:

cd docker
docker compose up --build -d

token/password = dunky

If not already selected, you can find Dunky kernel in the kernel list.

Remarks

This kernel is still in development and may have some bugs.
This extension works well together with the junity extension.

Issues?

If you encounter any issues, please open an issue on the GitHub repository.

dan1elt0m/dunky