SETL-Framework/setl

failed to setJobGroup

Closed this issue · 2 comments

conderls commented

Describe the bug

val factory = new SomeFactory()
factory.setJobGroup("new-group-name", "description")
pipeline.addStage(factory)

the JobGroup in the spark UI does not update.

Expected behavior
image
JobGroup and Description should be "new-group-name" and "description" respectively.

Environment (please complete the following information):

  • spark: 2.4.3
  • SETL: 1.0.0_RC1
qxzzxq commented

Hi @conderls thank you for your feedback. Indeed this is an issue. Currently your changes made by the call of setJobGroup will be overwritten when the stage is running (line 227 in workflow/Stage.scala).

A temporary workaround could be to call setJobGroup in the read() method of your factory. However, the limitation is that you may not be able to modify the job description.

We will fix this problem in the next release.

  • 👍1
stale commented

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.