trinodb/trino

Allow passing custom properties to Iceberg REST catalog

Opened this issue · 2 comments

devozerov commented

Proposal

Allow passing custom REST catalog properties in the config and session parameters:

iceberg.rest-catalog.additional-properties=prop1=val1,prop2=val2

SET SESSION <catalog_name>.rest-additional-properties=[ARRAY or MAP]['prop1=val1', 'prop2=val2']

Rationale

Iceberg REST specification allows properties to be passed to the REST catalog. Some of these properties are hard-coded and expected by the REST client (e.g., URL, warehouse). Other properties are custom and could be passed to the remote catalog, e.g., header.X-My-Custom-Header=value would translate into X-My-Custom-Header=value HTTP header in all REST catalog requests.

Since (1) multiple REST catalogs with different capabilities emerge and (2) even built-in Iceberg properties change over time (see RESTSessionCatalog history).

Both Spark and Flink have already solved this issue, allowing users to propagate custom properties without any restrictions. E.g.:

spark.sql.catalog.test_catalog.header.X-My-Custom-Header=value

The ability to pass custom catalog properties is important to ensure that users can always utilize the full power of the underlying REST catalog implementation, regardless of the Trino version used.

findinpath commented

@devozerov please detail a bit the use cases that you have in mind. Let's not implement a functionality based on the rationale that another engine happens to offer this specific feature.

  • 👍1
devozerov commented

For example, Polaris catalog has a notion of "realm":

Represents the elements of a REST request associated with routing to independent and isolated "universes".
This may include properties such as region, deployment environment (e.g. dev, qa, prod), and/or account.

The realm is passed via HTTP header header.realm. Spark and Flink users can use this feature, Trino's users can't. There are multiple catalogs emerging here and there, with various properties that they can utilize to provide additional user benefits.

Generally, we are talking about a part of Iceberg protocol that cannot be used by Trino users. Passing generic property bags around is a common approach for almost all Iceberg objects. This specific issue is concerned with REST catalog configuration that is specific for the catalog, but not for Trino.

Do you envision any problems if we add this feature as an additional config parameter?