Serialization issues
Closed this issue · 4 comments
Since flo 0.2.9 and introduction of forking and serialization I am running into issues testing and running flo tasks.
The current errors I get are:
java.lang.AssertionError: assertion failed: (List(import java.this.lang._),List())
scala.MatchError: None (of class scala.None$)
They seem related to:
- https://groups.google.com/forum/#!msg/akka-user/jvbbilXDKdA/giruRnnSCwAJ
- https://issues.scala-lang.org/browse/SI-9808
- https://groups.google.com/forum/m/#!topic/scala-user/IL_RfFj_yxE
- https://stackoverflow.com/questions/21687142/scala-match-on-optionobject-not-exhaustive-with-someo-and-none-in-akka-recei
My current hunch is that we might need to introduce chill and improve the kryo registrar configuration, similar to what scio does.
I created the ticket to raise awareness of the issue, ask for help and document my findings. I still need to bring the steps to reproduce the issue.
This should be resolved by #107
@labianchin is is this still an issue?
I have to update to newer version of flo and try it out. I might do this later this week.
I believe a likely scenario is that I won't find anymore issues, but still in the future it is possible some people will try to share non serializable objects between flo tasks.
So, I would suggest that we improve the documentation on how flo tasks run, how to troubleshoot serialization issues, known issues?, FAQ?,etc. before closing this task. It does need to be long and super complete documentation, maybe we can start with 4-5 sentences related to serialization, maybe pointing to this Github issue.
Now using flo 0.2.16 I have not experienced any serialization issues.