Flaky test: test_judge_comprehensive_grounding_evaluation has inconsistent completeness scores

[abrookins]

Problem Description

The test tests/test_llm_judge_evaluation.py::TestLLMJudgeEvaluation::test_judge_comprehensive_grounding_evaluation is flaky and exhibits inconsistent behavior across CI runs.

Test Details

File: tests/test_llm_judge_evaluation.py
Function: TestLLMJudgeEvaluation::test_judge_comprehensive_grounding_evaluation

Observed Behavior

• Test intermittently fails with assertion assert 0.2 >= 0.3
• Completeness scores vary between LLM evaluation runs
• Fails specifically on temporal grounding: 'next week' not resolved to specific dates
• Other aspects work correctly (pronoun resolution: 1.0, spatial grounding: 1.0)
• Inconsistent results across different Redis configurations

Root Cause

LLM-based evaluation tests are inherently non-deterministic due to:

• Variability in LLM response quality and consistency
• Model temperature and sampling affecting evaluation scores
• Dependency on external AI service reliability

Temporary Fix Applied

Commit: a781461 - "Fix flaky LLM evaluation test threshold"

• Lowered completeness threshold from 0.3 to 0.2
• This addresses the immediate CI failure but doesn't solve the underlying stability issue

Suggested Long-term Improvements

1. Add retry logic for flaky tests

   • Implement test retries for LLM-dependent evaluations
   • Use pytest-rerunfailures or custom retry decorators

2. Use mock responses for predictable testing

   • Mock LLM evaluation responses for deterministic results
   • Reserve real LLM tests for integration/manual testing

3. Implement test stability metrics

   • Track test success rates over time
   • Alert when stability drops below acceptable thresholds

4. Separate LLM evaluation tests

   • Make LLM evaluation tests optional in CI (env flag controlled)
   • Run as separate test suite or nightly builds
   • Keep core functionality tests deterministic

5. Improve evaluation robustness

   • Use multiple evaluation attempts and average scores
   • Implement confidence intervals for LLM evaluations
   • Add more specific temporal grounding test cases

Impact

• Intermittent CI failures blocking PRs
• False negatives reducing confidence in test suite
• Technical debt accumulating from threshold adjustments