`python-expr` one-liner seems to be a no-op
dovinmu opened this issue · 6 comments
The goal here is to extract out the string value of the first key in each item in the examples
list.
!format
https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/strategyqa/task.json
!read
!json-parse examples=examples
!!python-expr>key
list(examples[0]['target_scores'].keys())[0]
!python-foo
errors are getting eaten unless you use !debug
.
> /home/anja/git/aipl/aipl/interpreter.py(167)run_cmdlist()
-> if cmd.op.rankout is None:
(Pdb) p result
Error(linenum=6, opname='python_expr', exception=AssertionError())
Ah, so this is a bit tricky.
This is the signature:
def op_python_expr(aipl, row, expr:str):
But, the only input operand that exists is that expr
prompt. And it gets brought in as a Table and fails this assertion error:
> /home/anja/git/aipl/aipl/interpreter.py(276)prep_input()
-> assert isinstance(operand, LazyRow)
(Pdb) n
AssertionError
So, the problem is that python-expr
needs both a row, and an expr
. When it was the first !!
, there was no source row for it to pull from.
When I changed it to this:
1 !format
2 https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/strategyqa/task.json 3 !read 4 5 !json-parse>examples examples=examples
6 !python-expr>key
7 list(examples[0]['target_scores'].keys())[0]
8 !print
There seems to be a bug at the !json-parse
step:
- we already know that
examples=examples
will not result in an examples column >examples
does not work either, for reasons I do not fully understand.- Weirdly,
>examples>examples
did.
- Weirdly,
A note:
sometimes, we will get this Exception for Python ops:
Traceback (most recent call last):
File "/home/anja/anaconda3/envs/aipl/bin/aipl", line 8, in <module>
sys.exit(main())
File "/home/anja/git/aipl/aipl/main.py", line 87, in main
inputs = aipl.run(open(fn).read(), inputs)
File "/home/anja/git/aipl/aipl/interpreter.py", line 128, in run
return self.run_cmdlist(cmds, inputs)
File "/home/anja/git/aipl/aipl/interpreter.py", line 178, in run_cmdlist
raise AIPLException(f'AIPL Error (line {cmd.linenum} !{cmd.opname}): {e}') from e
File "/home/anja/git/aipl/aipl/__init__.py", line 39, in __str__
r.append(' ' + codestr.splitlines()[frame.lineno-1])
IndexError: list index out of range
Where it is a failure at the __repr__
. The reason it cannot be represented, is sometimes that there was an InnerPythonException
.
And then you need to use --debug
to see it:
File "/home/anja/git/aipl/aipl/table.py", line 84, in __getitem__
raise KeyError(k)
KeyError: 'target_scores'
I think this is as far as I am going to get with debugging this, because I ran into the fundamental issue:
Which is how !json-parse
processed the target_scores
.
We might want to break this up into sub-tasks:
- How do we find out earlier that a
!python-
Error is being hidden. (Default of!debug
). - What is expected for
!json-parse
to return, and how should it then be interacted with. (Right now, it flattens dictionaries.)