apache/beam

[Task]: Add utilities to easily implement portable dynamic destinations

ahmedabu98 opened this issue · 0 comments

ahmedabu98 commented

What needs to happen?

Dynamic destinations is a feature that allows users to write to multiple destinations using one IO instance. This is usually done via some sort of function that inspects the record and determines its destination. Currently, this feature is only supported with native use-cases (e.g. Python can use callables; Java can use SerializableFunctions or DynamicDestinations).

Of course, we can’t simply pass a whole function from one language to another, but we can extract the important bits and articulate them in a language-agnostic way

After discussion in the mailing list and design document, the agreed upon approach (proposal (1) in the doc) looks something like this:

image

PR #31807 is a proof of concept that includes the necessary utilities and a real implementation for portable dynamic destinations in IcebergIO. This PR will be broken down into multiple PRs to decouple concepts and make it easier to review.

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner