lanpa/tensorboardX

Failed tests in `tests/test_pr_curve.py`: cannot schedule new futures after interpreter shutdown

ndl opened this issue · 5 comments

ndl commented

Describe the bug

Running pytest on both the latest tagged version (2.4) and master results in failed tests in tests/test_pr_curve.py, specifically test_pr_purve and test_pr_purve_raw (test_smoke runs fine). The error message is RuntimeError: cannot schedule new futures after interpreter shutdown.

Minimal runnable code to reproduce the behavior

pytest . in master tensorboardX checkout.

Expected behavior

All enabled tests succeeding.

There are other tests failing apart of tests/test_pr_curve.py, BTW (tests/test_lint.py, tests/test_summary.py, tests/test_visdom.py and tests/test_writer.py) but each failure seems to be a separate issue.

Environment

Reproduced both in docker image for Ubuntu 22.04 and in FHSUserEnv environment on NixOS 21.11 + master nixpkgs.

pip list|grep -E "torch|proto|tensor"
protobuf                3.15.0
tensorboard             2.7.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.0
torch                   1.10.1
torchfile               0.1.0

Python environment

virtualenv 20.10.0 and Python 3.9.9 both on Ubuntu 22.04 and NixOS.

Additional context

The stack trace:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/var/tmp/tensorboardX/tensorboardX/event_file_writer.py", line 69, in flush
    self._py_recordio_writer.flush()
  File "/var/tmp/tensorboardX/tensorboardX/record_writer.py", line 187, in flush
    self._writer.flush()
  File "/var/tmp/tensorboardX/tensorboardX/record_writer.py", line 93, in flush
    s3.upload_fileobj(upload_buffer, bucket, path)
  File "/var/tmp/tensorboardx-env/lib/python3.9/site-packages/boto3/s3/inject.py", line 537, in upload_fileobj
    future = manager.upload(
  File "/var/tmp/tensorboardx-env/lib/python3.9/site-packages/s3transfer/manager.py", line 326, in upload
    return self._submit_transfer(
  File "/var/tmp/tensorboardx-env/lib/python3.9/site-packages/s3transfer/manager.py", line 500, in _submit_transfer
    self._submission_executor.submit(
  File "/var/tmp/tensorboardx-env/lib/python3.9/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/lib/python3.9/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
Mic92 commented

Tests can be fixed like this: #651 (comment)

  • 👍1
lanpa commented

Is that caused by python 3.9.9?

ndl commented

@Mic92 IIUC setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION = "python" in Nix derivation will fix the tests but won't change the behavior of the library in user applications, that is - the behavior of tests will be different from the "actual" behavior of the library as observed by the users (unless they set this variable in their environment as well).

If using python implementation is not critical for some other reason in this project, the cleaner approach is likely to change the goldens in tensorboardx test to match the cpp implementation?

lanpa commented

Hi @ndl , under ubuntu 22.04, I can't reproduce the cannot schedule new futures after interpreter shutdown error with test_pr_curve.py. But I can reproduce this error with tests/test_embedding.py. And It can be fixed by:
#657. Can you test this under nixos?

ndl commented

At current nixpkgs master branch I cannot reproduce test_pr_curve.py test failure either, but I do get the same error stack trace after tests finish (not attributed to any test in particular?)

After #657 is applied, as far as I can see there are no failures anymore - so I think this pull request fixes the issue indeed!

  • 🎉1