SETL-Framework/setl

how to delivery data from same Producer Factory with different params

conderls opened this issue · 2 comments

If I have a producer Factory whose result depends on param label,

class ProducerFactory(label: String) extends Factory[String] {
   override def deliveryId = "delivery_id"
   ...
  override def get() = label
}

I may use the data from ProducerFactory in another Factory by

class ConsumerFactory(label: String) extends Factory[String] {
   @Delivery(id = "delivery_id")
   private val label: String = ""
   ...
}

I can override deliveryId in ProducerFactory according to the param label, e.g. override def deliveryId = s"delivery_id_$label" for sure.

  1. I am wondering that, the ConsumerFactory is generic enough, If I have Producers with deliveryId: "delivery_id_a", "delivery_id_b", how can I using ConsumerFactory to consume data from these Producers? As I cannot set deliveryId in annotation according to the param label
   @Delivery(id = s"delivery_id_$label")
   private val label: String = "" // ERROR raised: annotation argument needs to be a constant;
  1. right now, I am creating new pipeline to use ProducerFactory and ConsumerFactory with fixed deliveryId; but how to clean the pipeline if there are some cached/persisted data.

Originally posted by @conderls in #263

Hello @conderls,
In order to generate the workflow based on relationships between factories and steps, the predictability of dependencies is mandatory, so dynamic delivery ids will broke this requirement.
I don't think this is something that we can acheive, at least not with the current design of the framework.
@qxzzxq if you want to add something ? if not you can close the ticket

Hi @conderls,
As @glc-mfelja says, the workflow must be deterministic at runtime. SETL does not support dynamic delivery. This is also due to the fact that Java/Scala's annotation does not support dynamically generated strings.