Potential planner bug in develop branch
johnynek opened this issue · 1 comments
johnynek commented
[info] - all optimization rules don't change results *** FAILED ***
[info] NullPointerException was thrown during property evaluation.
[info] Message: "None"
[info] Occurred when passed generated values (
[info] arg0 = WithDescriptionTypedPipe(TrappedPipe(WithDescriptionTypedPipe(MergedTypedPipe(WithDescriptionTypedPipe(EmptyTypedPipe,lVscjjUwdswuhslob3wcryitdoprmphqbl,false),WithDescriptionTypedPipe(EmptyTypedPipe,p8buqblzwenawcsgy8ckij7rndez4xlpcyJPwpzicbj9ct9hlekkzMhro5ugym9keiikxrjs7sutjuOhk,false)),org.scalacheck.Gen$R.map(Gen.scala:237),true),com.twitter.scalding.source.FixedTypedText(qayWlj3wtbwSlnkdhrgUTwilmjusWwsOdv7qdreth1933gtnuqzeythlyl6mbktjoimervxhhoavLbcyxB4duzrjxmk8b),Single(com.twitter.scalding.TupleGetter$IntGetter$@78d30dcc)),org.scalacheck.Gen$R.map(Gen.scala:237),true),
[info] arg1 = com.twitter.scalding.typed.OptimizationRules$FilterLocally$@342d6071.orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFlatMap$@12db1063).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMap$@41a47a5c).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFilter$@52cbaab4).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeWithOnComplete$@d3a7674).orElse(com.twitter.scalding.typed.OptimizationRules$RemoveDuplicateForceFork$@47a827b8).orElse(com.twitter.scalding.typed.OptimizationRules$ForceToDiskBeforeHashJoin$@253cd38b).orElse(com.twitter.scalding.typed.OptimizationRules$DeferMerge$@2a2d4392).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIterableIsEmpty$@747a0975).orElse(com.twitter.scalding.typed.OptimizationRules$HashToShuffleCoGroup$@726975a5)
[info] )
The stack that caused the error:
[info] Cause: java.lang.NullPointerException:
[info] at java.util.Objects.requireNonNull(Objects.java:203)
[info] at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
[info] at java.util.Arrays.asList(Arrays.java:3800)
[info] at cascading.pipe.Splice.<init>(Splice.java:280)
[info] at cascading.pipe.Splice.<init>(Splice.java:232)
[info] at cascading.pipe.Splice.<init>(Splice.java:192)
[info] at cascading.pipe.Splice.<init>(Splice.java:181)
[info] at cascading.pipe.Splice.<init>(Splice.java:428)
[info] at cascading.pipe.Merge.<init>(Merge.java:49)
[info] at com.twitter.scalding.typed.cascading_backend.CascadingBackend$$anon$3.$anonfun$toFunction$1(CascadingBackend.scala:268)
from commit: fa6eb87
oscar-stripe commented
Here's what looks like another hit of this:
[info] - all optimization rules don't change results *** FAILED ***
[info] NullPointerException was thrown during property evaluation.
[info] Message: "None"
[info] Occurred when passed generated values (
[info] arg0 = WithDescriptionTypedPipe(Mapped(WithDescriptionTypedPipe(CrossPipe(WithDescriptionTypedPipe(MergedTypedPipe(IterablePipe(List(2147483647)),IterablePipe(List(2147483647))),org.scalacheck.Gen$R.map(Gen.scala:237),true),IterablePipe(List(-1))),org.scalacheck.Gen$R.map(Gen.scala:237),true),<function1>),org.scalacheck.Gen$R.map(Gen.scala:237),true),
[info] arg1 = com.twitter.scalding.typed.OptimizationRules$AddExplicitForks$@1bc0b0e2.orElse(com.twitter.scalding.typed.OptimizationRules$ComposeFlatMap$@47811af2).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIterableIsEmpty$@48f48b19).orElse(com.twitter.scalding.typed.OptimizationRules$ForceToDiskBeforeHashJoin$@75940cb2).orElse(com.twitter.scalding.typed.OptimizationRules$EmptyIsOftenNoOp$@6bcde542).orElse(com.twitter.scalding.typed.OptimizationRules$ComposeMapFlatMap$@24543443).orElse(com.twitter.scalding.typed.OptimizationRules$FilterKeysEarly$@4894cf93).orElse(com.twitter.scalding.typed.OptimizationRules$HashToShuffleCoGroup$@1db0bbaf).orElse(com.twitter.scalding.typed.OptimizationRules$DescribeLater$@1953aaae)
[info] )
which has come up on another branch
Will add a test for that one too.