Unreliable tests.
mvaled opened this issue · 4 comments
mvaled commented
Two consecutive runs of the tests fail with different results. I'm using tox with a bare python 2.7 configuration:
[testenv:bare27]
basepython=python2.7
The first run of tox -e bare27 gets errors and failures:
======================================================================
ERROR: test_alias_unalias_command (test-cmd-alias.TestAliasCommand)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-cmd-alias.py", line 22, in setUp
d = Mdebugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
ERROR: test_completion (test-completion.TestCompletion)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-completion.py", line 33, in test_completion
self.dbgr = Mdebugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
ERROR: test_info_file (test-info-files.TestInfoFile)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-info-files.py", line 46, in test_info_file
d = Mdebugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
ERROR: test_list_command (test-list.TestListCommand)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-list.py", line 60, in test_list_command
d = debugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
ERROR: test_pdef (test-pdef.TestPDef)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-pdef.py", line 27, in test_pdef
d = debugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
ERROR: test_pr (test-pr.TestP)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-pr.py", line 27, in test_pr
d = debugger.Debugger()
File "/home/manu/src/github.com/languages-compilers/trepan/trepan/debugger.py", line 331, in __init__
self.sigmgr = Msig.SignalManager(self)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 181, in __init__
self.action('SIGINT stop print nostack nopass')
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 378, in action
return self.check_and_adjust_sighandler(signame, self.sigs)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 259, in check_and_adjust_sighandler
self._orig_set_signal(signum, self.sigs[signame].handle)
File "/home/manu/src/github.com/languages-compilers/trepan/.tox/bare27/lib/python2.7/site-packages/trepan/lib/sighandler.py", line 225, in set_signal_replacement
self.sigs[signame].pass_along = True
KeyError: 'ITIMER_PROF'
======================================================================
FAIL: test_lookup_signame_signum (test-lib-sig.TestLibSigHandle)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-lib-sig.py", line 47, in test_lookup_signame_signum
self.assertEqual(signum, Msig.lookup_signum(signame))
AssertionError: 0 != None
'0 != None' = '%s != %s' % (safe_repr(0), safe_repr(None))
'0 != None' = self._formatMessage('0 != None', '0 != None')
>> raise self.failureException('0 != None')
----------------------------------------------------------------------
Ran 67 tests in 0.468s
FAILED (errors=6, failures=1)
The second run now shows a single failure (no changes in the code):
======================================================================
FAIL: test_lookup_signame_signum (test-lib-sig.TestLibSigHandle)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/manu/src/github.com/languages-compilers/trepan/test/unit/test-lib-sig.py", line 47, in test_lookup_signame_signum
self.assertEqual(signum, Msig.lookup_signum(signame))
AssertionError: 0 != None
'0 != None' = '%s != %s' % (safe_repr(0), safe_repr(None))
'0 != None' = self._formatMessage('0 != None', '0 != None')
>> raise self.failureException('0 != None')
----------------------------------------------------------------------
Ran 67 tests in 0.501s
FAILED (failures=1)
A third execution all tests passed; a fourth went back to 6 errors and 1 failure. The 5th and 6th gave a single failure. The 7th passed.
Basically I cannot trust the tests.
birkenfeld commented
Basically I cannot trust the tests.
Please don't exaggerate. This only happens for me using tox. Running the test suite with make test works fine.
I'm looking at the tox-induced failures.
mvaled commented
Sorry if I seemed over-reacting.
The make test fails if the tracer and other dependencies are not
installed (which I didn't at first):
$ make test
python ./setup.py nosetests
running nosetests
running egg_info
writing requirements to trepan.egg-info/requires.txt
writing trepan.egg-info/PKG-INFO
writing namespace_packages to trepan.egg-info/namespace_packages.txt
writing top-level names to trepan.egg-info/top_level.txt
writing dependency_links to trepan.egg-info/dependency_links.txt
writing entry points to trepan.egg-info/entry_points.txt
reading manifest file 'trepan.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'trepan.egg-info/SOURCES.txt'
Searching for tracer>=0.3.2
Reading http://localhost:5001/simple/
Best match: tracer 0.3.2
Downloading http://localhost:5001/d/tracer-0.3.2.tar.gz
Processing tracer-0.3.2.tar.gz
Writing /tmp/easy_install-Dl66tU/tracer-0.3.2/setup.cfg
Running tracer-0.3.2/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-Dl66tU/tracer-0.3.2/egg-dist-tmp-YIZzgX
Traceback (most recent call last):
File "./setup.py", line 41, in <module>
zip_safe = zip_safe)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/manu/.local/lib/python2.7/site-packages/nose/commands.py",
line 144, in run
self.distribution.install_requires)
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 287, in
fetch_build_eggs
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 631, in resolve
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 874, in
best_match
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 886, in obtain
File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 338, in
fetch_build_egg
File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
613, in easy_install
File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
643, in install_item
File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
833, in install_eggs
File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
1055, in build_and_install
File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
1040, in run_setup
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 68, in
run_setup
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 114,
in run
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 67, in
runner
File "build/bdist.linux-x86_64/egg/setuptools/sandbox.py", line 43, in
_execfile
File "/tmp/easy_install-Dl66tU/tracer-0.3.2/setup.py", line 8, in <module>
from file __pkginfo__.py.
ImportError: cannot import name package_dir
make: *** [test-unit] Erreur 1
After installing all the dependencies it works.
Sorry again.
birkenfeld commented
No worries! Of course running tox should work when there is even a tox.ini configuration :) And you have uncovered a real bug in signal handling.
I usually use a virtualenv per project and pip install -r requirements.txt to prepare running tests.
mvaled commented
I used to have virtualenvs per project. But since long, I've moved to
buildout to avoid dealing with too many virtualenvs; but I keep using
tox for testing since it manages the virtualenvs by itself.