oughtinc/ice

Trace bug when using Pandas objects in recipes

brachbach opened this issue · 0 comments

The trace can't handle dataframes and series well

Repro:

  1. pull placebo_qa_trace_broken in ice-ought-private
  2. run python ought/eval/eval_paper_qa_vs_gs.py --recipe-to-run ought/placebo/placebo_keyword_baseline.py:placebo_keyword --gs-df /Users/benjaminrachbach/ought/ice/gold_standards/gold_standards.csv --splits validation --question-short-name placebo

The recipe runs, but the trace will error:

... 
  File "/Users/benjaminrachbach/ought/ice/ice/trace.py", line 351, in get_strings
    result = _get_first_descendant(value)
  File "/Users/benjaminrachbach/ought/ice/ice/trace.py", line 374, in _get_first_descendant
    if value:
  File "/Users/benjaminrachbach/ought/ice-ought-private/.venv/lib/python3.10/site-packages/pandas/core/generic.py", line 1527, in __nonzero__
    raise ValueError(
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

this seems to be because _get_first_descendant is at some point checking the truth value of one of the dataframes used/outputted in the recipe