Non-deterministic unit test failure

Question

Non-deterministic unit test failure

pmonks opened this issue 6 years ago · 0 comments

When running the tests locally on JDK 8 (i.e. with bundled JavaFX), I periodically see the following error, and a similar error has shown up on TravisCI too.

I don't seem to have the same issue on JDK 11 / OpenJFX 11, though it has occurred on TravisCI with that version combo.

I haven't discovered a pattern to its occurrence - often just rerunning the tests will succeed (both locally and on TravisCI). This is especially puzzling on TravisCI, since the container the tests run in there is thrown away between every single test run, significantly reducing the possibility that some kind of state is being preserved between successive runs of the tests.

$ lein do clean, version, test, jar
Leiningen 2.8.1 on Java 1.8.0_192 Java HotSpot(TM) 64-Bit Server VM

lein test fn-fx.diff-test

lein test fn-fx.fx-dom-test

lein test :only fn-fx.fx-dom-test/static-member-tests

ERROR in (static-member-tests) (fx_dom_test.clj:10)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.AssertionError: Assert failed: comp
 at fn_fx.fx_dom_test$get_prop.invokeStatic (fx_dom_test.clj:10)
    fn_fx.fx_dom_test$get_prop.invoke (fx_dom_test.clj:10)
    fn_fx.fx_dom_test$fn__4061.invokeStatic (fx_dom_test.clj:116)
    fn_fx.fx_dom_test/fn (fx_dom_test.clj:99)
    clojure.test$test_var$fn__9209.invoke (test.clj:716)
    clojure.test$test_var.invokeStatic (test.clj:716)
    clojure.test$test_var.invoke (test.clj:707)
    clojure.test$test_vars$fn__9235$fn__9240.invoke (test.clj:734)
    clojure.test$default_fixture.invokeStatic (test.clj:686)
    clojure.test$default_fixture.invoke (test.clj:682)
    clojure.test$test_vars$fn__9235.invoke (test.clj:734)
    clojure.test$default_fixture.invokeStatic (test.clj:686)
    clojure.test$default_fixture.invoke (test.clj:682)
    clojure.test$test_vars.invokeStatic (test.clj:730)
    clojure.test$test_all_vars.invokeStatic (test.clj:736)
    clojure.test$test_ns.invokeStatic (test.clj:757)
    clojure.test$test_ns.invoke (test.clj:742)
    user$eval233$fn__294.invoke (form-init3321944168700773224.clj:1)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:659)
    clojure.core$apply.invoke (core.clj:652)
    leiningen.core.injected$compose_hooks$fn__163.doInvoke (form-init3321944168700773224.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:657)
    clojure.core$apply.invoke (core.clj:652)
    leiningen.core.injected$run_hooks.invokeStatic (form-init3321944168700773224.clj:1)
    leiningen.core.injected$run_hooks.invoke (form-init3321944168700773224.clj:1)
    leiningen.core.injected$prepare_for_hooks$fn__168$fn__169.doInvoke (form-init3321944168700773224.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:29)
    clojure.lang.RestFn.invoke (RestFn.java:408)
    clojure.core$map$fn__5587.invoke (core.clj:2747)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1785)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invokeStatic (core.clj:659)
    clojure.test$run_tests.invokeStatic (test.clj:767)
    clojure.test$run_tests.doInvoke (test.clj:767)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:657)
    clojure.core$apply.invoke (core.clj:652)
    user$eval233$fn__306$fn__339.invoke (form-init3321944168700773224.clj:1)
    user$eval233$fn__306$fn__307.invoke (form-init3321944168700773224.clj:1)
    user$eval233$fn__306.invoke (form-init3321944168700773224.clj:1)
    user$eval233.invokeStatic (form-init3321944168700773224.clj:1)
    user$eval233.invoke (form-init3321944168700773224.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:7062)
    clojure.lang.Compiler.eval (Compiler.java:7052)
    clojure.lang.Compiler.load (Compiler.java:7514)
    clojure.lang.Compiler.loadFile (Compiler.java:7452)
    clojure.main$load_script.invokeStatic (main.clj:278)
    clojure.main$init_opt.invokeStatic (main.clj:280)
    clojure.main$init_opt.invoke (main.clj:280)
    clojure.main$initialize.invokeStatic (main.clj:311)
    clojure.main$null_opt.invokeStatic (main.clj:345)
    clojure.main$null_opt.invoke (main.clj:342)
    clojure.main$main.invokeStatic (main.clj:424)
    clojure.main$main.doInvoke (main.clj:387)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:702)
    clojure.main.main (main.java:37)

lein test :only fn-fx.fx-dom-test/test-basic-component-properties

FAIL in (test-basic-component-properties) (fx_dom_test.clj:23)
expected: (= (get-prop (clojure.core/deref root) :title) "Hello")
  actual: (not (= nil "Hello"))

Ran 9 tests containing 59 assertions.
1 failures, 1 errors.
Tests failed.