Exception during exception handling
Markus28 opened this issue · 8 comments
Hey,
I have some issues with the exception handling in sacred: Consider the slightly modified "hello world" example from the docs which throws an exception:
main.py:
from sacred import Experiment
ex = Experiment('hello_config')
@ex.config
def my_config():
recipient = "world"
message = "Hello %s!" % recipient
@ex.automain
def my_main(message):
raise ValueError("Some Error")
Now, python main.py produces the following output:
WARNING - hello_config - No observers have been added to this run
INFO - hello_config - Running command 'my_main'
INFO - hello_config - Started
ERROR - hello_config - Failed after 0:00:00!
Traceback (most recent call last):
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/experiment.py", line 312, in run_commandline
return self.run(
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/experiment.py", line 276, in run
run()
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/run.py", line 238, in __call__
self.result = self.main_function(*args)
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/config/captured_function.py", line 42, in captured_function
result = wrapped(*args, **kwargs)
File "/home/markus/test/sacred_test/main.py", line 12, in my_main
raise ValueError("Some Error")
ValueError: Some Error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/markus/test/sacred_test/main.py", line 11, in <module>
def my_main(message):
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/experiment.py", line 190, in automain
self.run_commandline()
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/experiment.py", line 347, in run_commandline
print_filtered_stacktrace()
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/utils.py", line 493, in print_filtered_stacktrace
print(format_filtered_stacktrace(filter_traceback), file=sys.stderr)
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/utils.py", line 528, in format_filtered_stacktrace
return "".join(filtered_traceback_format(tb_exception))
File "/home/markus/test/sacred_test/env/lib64/python3.9/site-packages/sacred/utils.py", line 568, in filtered_traceback_format
current_tb = tb_exception.exc_traceback
AttributeError: 'TracebackException' object has no attribute 'exc_traceback'
i.e. there is another exception during the handling of the ValueError.
This was produced on a fresh virtual environment with Python 3.9.7 and Sacred 0.8.2. The behavior is reproducible across several machines and was also observed by a coworker. It is also mentioned here: https://stackoverflow.com/questions/67014839/attributeerror-tracebackexception-object-has-no-attribute-exc-trackback-err and here: ray-project/ray#5181
This issue seems to be relatively new, but I cannot track down, what versions/changes introduced this behavior.
This is a major issue for us because we intentionally raise exceptions to log and schedule restarts of the experiments.
I would be grateful for some help!
Best,
Markus
Yes, I think that that commit fixed the exception handling for Python 3.9. It could be time for a new release on pypi with the fixes for Python 3.9.
@thequilo I'm also facing this issue.
Do you recon a new release will be made soon?
Do you recommend to pip install Sacred's main branch, for now?
Hi @lucasdavid! Yes and yes (see #857). The reason for the delay in the releases were some failing testcases caused by an update of pymongo while the corresponding update of mongomock was not yet there. mongomock released a new version a week ago, so now these testcases can be fixed.
Ok, thank you very much!
Any update, I am still getting the same exception for Python 3.9?
Same exception, same problem. This does not seem to be fixed.
Same exception, same problem in python3.8.