how to delivery data from same Producer Factory with different params
conderls opened this issue · 2 comments
If I have a producer Factory whose result depends on param label
,
class ProducerFactory(label: String) extends Factory[String] {
override def deliveryId = "delivery_id"
...
override def get() = label
}
I may use the data from ProducerFactory in another Factory by
class ConsumerFactory(label: String) extends Factory[String] {
@Delivery(id = "delivery_id")
private val label: String = ""
...
}
I can override deliveryId
in ProducerFactory according to the param label
, e.g. override def deliveryId = s"delivery_id_$label"
for sure.
- I am wondering that, the ConsumerFactory is generic enough, If I have Producers with
deliveryId
:"delivery_id_a", "delivery_id_b"
, how can I using ConsumerFactory to consume data from these Producers? As I cannot set deliveryId in annotation according to the paramlabel
@Delivery(id = s"delivery_id_$label")
private val label: String = "" // ERROR raised: annotation argument needs to be a constant;
- right now, I am creating new pipeline to use ProducerFactory and ConsumerFactory with fixed
deliveryId
; but how to clean the pipeline if there are some cached/persisted data.
Hello @conderls,
In order to generate the workflow based on relationships between factories and steps, the predictability of dependencies is mandatory, so dynamic delivery ids will broke this requirement.
I don't think this is something that we can acheive, at least not with the current design of the framework.
@qxzzxq if you want to add something ? if not you can close the ticket
Hi @conderls,
As @glc-mfelja says, the workflow must be deterministic at runtime. SETL does not support dynamic delivery. This is also due to the fact that Java/Scala's annotation does not support dynamically generated strings.