IBMStreams/streamsx.kafka

KafkaConsumer: support submission time parameter for multiple topics

Opened this issue · 1 comments

The KafkaConsumer's topic operator parameter is of type rstring with cardinality -1, which allows to specify multiple topics to subscribe, like this:

stream <MessageType.StringMessage> Consumed = KafkaConsumer() {
    param
        topic: "t1", "t2";
        ...
}

There seems to be no way, to use a submission time parameter for multiple values.
The SPL function getSubmissionTimeValue returns always a single rstring value, which would be interpreted as a single topic, i.e. when an applicatioin is submitted with -P topics_param=t1,t2, the operator tries to subscribe to the single topic "t1,t2" instead of two topics.

The SPL function getSubmissionTimeListValue returns a list of rstrings, which is incompatible with the generated operator model:
CDISP0048E ERROR: The topic operator parameter requires values of the rstring type, but values of the list<rstring> type are specified.

A workaround must be implemented here:

Try to split given topic(s) at a comma character, subscribe to the tokens.
Pro: No new operator parameter
Con: topics with a comma in it cannot be subscribed. Currently not an issue as a comma is not (yet) a valid character within a topic. Chances that a comma will be valid in future are very low.

fixed in v3.2.2