Failing tests
zupo opened this issue · 9 comments
Tests on main branch fail on my local (macbook m2):
____________________________ test_multiline_command ____________________________
def test_multiline_command() -> None:
"""Test using `> ` to extend a command across more than one line."""
runner = CliRunner()
result = runner.invoke(tesh, "src/tesh/tests/fixtures/multiline_command.md")
> assert result.exit_code == 0
E assert 1 == 0
E + where 1 = <Result SystemExit(1)>.exit_code
➜ tesh git:(main) poetry run tesh src/tesh/tests/fixtures/multiline_command.md
📄 Checking src/tesh/tests/fixtures/multiline_command.md
✨ Running readme-example ❌ Failed
Command: echo "Hello from" \
"another" \
"line!"
Expected:
Hello from another line!
Got:
> > Hello from another line!
Taking you into the shell ...
Enter `!!` to rerun the last command.
$
README.me fails as well:
➜ tesh git:(fix/multiline_output_macs) make tesh
📄 Checking CHANGELOG.md
📄 Checking README.md
✨ Running hello ✅ Passed
✨ Running multiple_blocks ✅ Passed
✨ Running ignore ❌ Failed
Command: echo "Hello from" \
"another" \
"line!"
Expected:
Hello from another line!
Got:
> > Hello from another line!
Taking you into the shell ...
Enter `!!` to rerun the last command.
$
Have to get off the ferry, this is how far I got:
[7] > /Users/zupo/work/tesh/src/tesh/test.py(167)compare_outputs()
-> if not fnmatch.fnmatch(actual_output, expected_output):
(Pdb++) actual_output
'> > Hello from another line!'
(Pdb++) expected_output
'Hello from another line!'
Hmm, looks like it's to do with line continuation markers, based on the way you have > > Hello from another line! when it expects no > characters. I just tried on a Mac and I also get failures, but seemingly different ones to you. Mine are timing out, so pexpect is failing to match for some reason. I'll have a look and see if I can see what's going on.
Not sure if it's related to your problem but my errors seem to be caused by the way pexpect is translating multi-byte unicode codepoints in shell commands.
In [26]: shell = pexpect.spawn("bash --norc --noprofile", encoding="utf-8",
...: env={"PS1": "$ ", "PATH": os.environ["PATH"], "HOME": os.getcwd()},
...: dimensions=(24, 1000))
In [27]: shell.sendline('echo "foo ✨"')
Out[27]: 15
In [28]: shell.expect(r"\$ ")
Out[28]: 0
In [29]: shell.before
Out[29]: ''
In [30]: shell.expect(r"\$ ")
Out[30]: 0
In [31]: shell.before
Out[31]: 'echo "foo \\342\\234\\250"\r\nfoo ✨\r\n'
It's turning them into octal ecapes which fail to match the actual character in the command when tesh tries to match the command. This is causing simple.md to fail for me.
On Linux I get this:
In [1]: import os
In [2]: import pexpect
In [3]: shell = pexpect.spawn("bash --norc --noprofile", encoding="utf-8",
...: env={"PS1": "$ ", "PATH": os.environ["PATH"], "HOME": os.getcwd()},
...: dimensions=(24, 1000))
In [4]: shell.sendline('echo "foo ✨"')
Out[4]: 15
In [5]: shell.expect(r"\$ ")
Out[5]: 0
In [6]: shell.expect(r"\$ ")
Out[6]: 0
In [7]: shell.before
Out[7]: 'echo "foo ✨"\r\nfoo ✨\r\n'
My issue does seem to be different to yours. I've found using pexpect interactively in a shell like ipython to be a good way to work out what's going on when it's not working as expected.
I suggest you try matching the multiline command in an interactive session to see where it's screwing up. Here's how it should work matching the multiline example from the readme. (I notice that it should be in a different tesh-session!)
In [1]: from tesh.extract import extract
In [2]: with open('README.md') as f:
...: sessions = extract(f)
...:
In [3]: sessions
Out[3]:
[ShellSession(lines=['$ echo "Hello World!"\n', 'Hello World!\n'], blocks=[], id_='hello', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ export NAME=Earth\n', '\n', '$ echo "Hello $NAME!"\n', 'Hello Earth!\n'], blocks=[], id_='multiple_blocks', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ echo "Hello from Space!"\n', 'Hello ... Space!\n', '$ printf "Hello \\nthere \\nfrom \\nSpace!"\n', 'Hello\n', '...\n', 'Space!\n', '$ echo "Hello from" \\\n', '> "another" \\\n', '> "line!"\n', 'Hello from another line!\n'], blocks=[], id_='ignore', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ false\n', '\n', '$ true\n', '\n'], blocks=[], id_='exitcodes', ps1=None, setup=None, exitcodes=[1, 0], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ echo "Hello $NAME!"\n', 'Hello Gaea!\n'], blocks=[], id_='setup', ps1=None, setup='readme.sh', exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ PS1="(foo) $ "\n', '\n', '\n', '(foo) $ echo "hello"\n', 'hello\n'], blocks=[], id_='prompt', ps1='(foo) $', setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ uname\n', '...Darwin...\n'], blocks=[], id_='platform', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False),
ShellSession(lines=['$ chmod +x foo.sh\n', '\n', '$ ./foo.sh\n', 'foo\n'], blocks=[], id_='fixture', ps1=None, setup=None, exitcodes=[], fixtures=[Fixture(filename='foo.sh', contents='echo "foo"\n')], timeout=30, long_running=False),
ShellSession(lines=['$ sleep 1\n', '\n'], blocks=[], id_='timeout', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=3, long_running=False),
ShellSession(lines=['$ nmap 1.1.1.1\n', 'Starting Nmap ...\n'], blocks=[], id_='long-running', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=1, long_running=True)]
In [4]: ignore, = [s for s in sessions if s.id_ == 'ignore']
In [5]: from tesh.extract import extract_blocks
In [7]: extract_blocks(ignore, False)
In [8]: ignore
Out[8]: ShellSession(lines=['$ echo "Hello from Space!"\n', 'Hello ... Space!\n', '$ printf "Hello \\nthere \\nfrom \\nSpace!"\n', 'Hello\n', '...\n', 'Space!\n', '$ echo "Hello from" \\\n', '> "another" \\\n', '> "line!"\n', 'Hello from another line!\n'], blocks=[Block(command='echo "Hello from Space!"', output=['Hello ... Space!'], prompt='$ '), Block(command='printf "Hello \\nthere \\nfrom \\nSpace!"', output=['Hello', '...', 'Space!'], prompt='$ '), Block(command='echo "Hello from" \\\n "another" \\\n "line!"', output=['Hello from another line!'], prompt='$ ')], id_='ignore', ps1=None, setup=None, exitcodes=[], fixtures=[], timeout=30, long_running=False)
In [9]: len(ignore.blocks)
Out[9]: 3
In [10]: ignore.blocks[2]
Out[10]: Block(command='echo "Hello from" \\\n "another" \\\n "line!"', output=['Hello from another line!'], prompt='$ ')
In [11]: import pexpect
In [13]: import os
In [14]: shell = pexpect.spawn("bash --norc --noprofile", encoding="utf-8",
...: env={"PS1": "$ ", "PATH": os.environ["PATH"], "HOME": os.getcwd()},
...: dimensions=(24, 1000))
In [15]: shell.expect(r"\$ ")
Out[15]: 0
In [16]: shell.sendline(ignore.blocks[2].command)
Out[16]: 44
In [17]: cmd_lines = list(ignore.blocks[2].command.splitlines())
In [18]: cmd_lines
Out[18]: ['echo "Hello from" \\', ' "another" \\', ' "line!"']
In [19]: shell.expect_exact(cmd_lines[0])
Out[19]: 0
In [20]: shell.expect_exact(cmd_lines[1])
Out[20]: 0
In [21]: shell.expect_exact(cmd_lines[2])
Out[21]: 0
In [22]: shell.expect(r"\$ ")
Out[22]: 0
In [23]: shell.before
Out[23]: '\r\nHello from another line!\r\n'
I expect you'll find line 23 is different for you. See how mine does not have continuation markers in the output, but yours appears to.
Something I noticed when working on this before was that pexpect has a higher-level API called replwrap that handles this more low-level matching of the prompt and command. It could be that if we used that it would handle whatever the shell difference is that's going on here. https://pexpect.readthedocs.io/en/stable/api/replwrap.html
Yep, there's > > in shell.before:
>>> len(ignore.blocks)
3
>>> ignore.blocks[2]
Block(command='echo "Hello from" \\\n "another" \\\n "line!"', output=['Hello from another line!'], prompt='$ ')
>>> import pexpect
>>> import os
>>> shell = pexpect.spawn("bash --norc --noprofile", encoding="utf-8",
... env={"PS1": "$ ", "PATH": os.environ["PATH"], "HOME": os.getcwd()},
... dimensions=(24, 1000))
>>> shell.expect(r"\$ ")
0
>>> shell.sendline(ignore.blocks[2].command)
44
>>> cmd_lines = list(ignore.blocks[2].command.splitlines())
>>> cmd_lines
['echo "Hello from" \\', ' "another" \\', ' "line!"']
>>> shell.expect_exact(cmd_lines[0])
0
>>> shell.expect_exact(cmd_lines[1])
0
>>> shell.expect_exact(cmd_lines[2])
0
>>> shell.expect(r"\$ ")
0
>>> shell.before
'\r\n> > Hello from another line!\r\n'
>>>