ultimate-pa/ultimate

Fix regression tests

Opened this issue · 0 comments

On the Nightly build 405 we had 320 failed regression tests. To improve our workflow we should avoid failures of regression tests in the future.

Issues with test-suites/verdicts:

Tests that succeed locally, but fail in Jenkins-Build:

Issues with AutomataLibraryRegressionTestSuite:

Issues with ReqCheckerRegressionTestSuite:

Other Issues:

  • PdrAutomizer: SIMPLIFY_QUICK does not support simplification with respect to context, should we use another simplification technique?
  • PdrAutomizer: UnsupportedOperationException: Solver returned unknown
  • recursive-CallABAB_incorrect.bpl (abstract interpretation): Looks like an actual unsoundness.
  • overflow.c: We currently do not add any in-range-assumptions for parameters. This can be unsound, if we are in library mode (i.e. the entry-method is not declared). Should we adapt this or is the library mode not important enough for us? An easy fix would be to always assume that the arguments are in range, but not sure, if this is desired.
  • Enum03-Unsafe.c: ClassCastException: class CEnum cannot be cast to class CPrimitive in TypeSizes.extractIntegerValue
  • Some Float tests with MathSAT (e.g. this): SMTLIBException: The CNF conversion does not support quantifiers
  • ctrans-float-round.c: unknown symbol: roundNearestTiesToAway (with two strategies); Problem with MathSAT?
  • float22_simplified.c, ctrans-float-union.c: Sort Array not declared
  • ctrans-float-union.c: The translation of unions is unsound.
  • SubwordAccess.c: Looks like an actual unsoundness, how should we proceed with that?
  • Bug_ProcedureDNF.c (LTL): Assertion error in "old" map elimination
  • Some tests of ChcRegressionTestSuite fail because of issues in TreeAutomizer or because external solvers (like Z3) time out.
  • LTLAutomizer: AssertionError: No corresponding entry node for exit node with procedure main
  • UpAndDownGlobalReals.bpl: AssertionError in SMTInterpol (Interpolator$MixedLAInterpolator::convert)
  • builtin_ffs.c (with bitvector settings): IllegalArgumentException: incompatible types LONG INT
  • structFlexibleArray01.c: NumberFormatException: For input string: "18446744073708317049"
  • flexible_array.c: Unsoundness with bitvector translation

Unsupported features:

  • SyntaxSupportMixedIntReal2.bpl: PolynomialRelation does not support mixed sorts. The problem is that we do not really support ranking functions for real variables, since most constants and variables created in the termination analysis have the sort int. Therefore we often create terms with inconsistent sorts.
  • multipleCallSuccessors-01.bpl: State has more than one call successor
  • C2BoogieRegressionTestSuite: All the failing tests are currently unsupported cases, do we want to keep those as regression tests?
  • Some tests of the RegressionTestSuite (e.g this) also fail, because we don't support a specific feature.
  • ConstArray.bpl: SMTLIBException: Const is only supported for infinite index sort (with three settings)
  • InParamRenaming.c: Issue with renaming parameters that are used in ACSL
  • NonterminatingForLoopSafe.c: Some settings don't support non-linear arithmetic, we could however just simplify this example, the intention would remain the same.

Tests fail with unknown/timeout:

  • Several SMT tests are too hard for the underlying solver and should not run in regression tests.
  • Most of the other tests (e.g. from RegressionTestSuite) simply always fail, because we say unknown or have a timeout (at least for some of the settings/toolchains). We should try to fix them and otherwise find a way to ignore these failures in the test result.