KafkaConsumer: support submission time parameter for multiple topics
Opened this issue · 1 comments
The KafkaConsumer's topic operator parameter is of type rstring
with cardinality -1, which allows to specify multiple topics to subscribe, like this:
stream <MessageType.StringMessage> Consumed = KafkaConsumer() {
param
topic: "t1", "t2";
...
}
There seems to be no way, to use a submission time parameter for multiple values.
The SPL function getSubmissionTimeValue
returns always a single rstring
value, which would be interpreted as a single topic, i.e. when an applicatioin is submitted with -P topics_param=t1,t2
, the operator tries to subscribe to the single topic "t1,t2" instead of two topics.
The SPL function getSubmissionTimeListValue
returns a list of rstrings, which is incompatible with the generated operator model:
CDISP0048E ERROR: The topic operator parameter requires values of the rstring type, but values of the list<rstring> type are specified.
A workaround must be implemented here:
Try to split given topic(s) at a comma character, subscribe to the tokens.
Pro: No new operator parameter
Con: topics with a comma in it cannot be subscribed. Currently not an issue as a comma is not (yet) a valid character within a topic. Chances that a comma will be valid in future are very low.
fixed in v3.2.2