progress stalled; perhaps related to too many open files
wideglide opened this issue · 1 comments
FACT version
Environment
- Ubuntu 22.04.4 LTS
- (venv) Python 3.10.12
Steps to reproduce
- processing firmware images
- ASUS RT-AX82U v. 3.0.0.4.388.22158
- TP-Link Archer C59(US)_V2_220113.zip
Observed Behavior
Processing stalled.
Expeced Behavior
Processed firmware image normally.
Installation logs
no issues during install.
Backend logs
fact_main_backend.log
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 2df36886966bb3f3ce60c849fed95e7ff02d6a0ae1c2d8889b8442cfaddd5790_5392
Traceback (most recent call last):
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
self.work_thread(task, container)
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
self.cleanup()
File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
_rmtree(name, onerror=onerror)
File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
onerror(os.scandir, path, sys.exc_info())
File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmprpukq11b/tmpbm_gz_v9/reports'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of cc528127f922195150ad66dd7254c6275392a06beeab9f8a7905d2ec3a8e2562_7756
Traceback (most recent call last):
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
self.work_thread(task, container)
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
self.cleanup()
File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
_rmtree(name, onerror=onerror)
File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
onerror(os.scandir, path, sys.exc_info())
File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgs5tv66w/tmp22hy3plt/reports'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 0ecf29ed3821a55d6dfa80f75ad6777e64f71d47a305bbe9fc50979352dba0f8_1248
Traceback (most recent call last):
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 197, in work_thread
extracted_objects = self.unpacker.unpack(task, tmp_dir, container)
File "/home/kyle/FACT_core/src/unpacker/unpack.py", line 43, in unpack
extracted_files = self.extract_files_from_file(current_fo.file_path, tmp_dir, container)
File "/home/kyle/FACT_core/src/unpacker/unpack_base.py", line 35, in extract_files_from_file
shutil.copy2(file_path, str(Path(tmp_dir, 'input', Path(file_path).name)))
File "/usr/lib/python3.10/shutil.py", line 434, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.10/shutil.py", line 256, in copyfile
with open(dst, 'wb') as fdst:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgu0kfqtb/tmpmtxr_4gt/input/0ecf29ed3821a55d6dfa80f75ad6777e64f71d47a305bbe9fc50979352dba0f8_1248'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 187, in _work_thread_wrapper
self.work_thread(task, container)
File "/home/kyle/FACT_core/src/scheduler/unpacking_scheduler.py", line 195, in work_thread
with TemporaryDirectory(dir=container.tmp_dir.name) as tmp_dir:
File "/usr/lib/python3.10/tempfile.py", line 1008, in __exit__
self.cleanup()
File "/usr/lib/python3.10/tempfile.py", line 1012, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "/usr/lib/python3.10/tempfile.py", line 994, in _rmtree
_rmtree(name, onerror=onerror)
File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/lib/python3.10/shutil.py", line 633, in _rmtree_safe_fd
onerror(os.scandir, path, sys.exc_info())
File "/usr/lib/python3.10/shutil.py", line 629, in _rmtree_safe_fd
with os.scandir(topfd) as scandir_it:
OSError: [Errno 24] Too many open files: '/tmp/fact-docker-mount-base-dir/tmpgu0kfqtb/tmpmtxr_4gt'
[2024-03-19 10:32:11][unpacking_scheduler][ERROR]: Exception occurred during unpacking of 8401139708fdc9407e43b05d0df712fc89061946fab799917e11937032745f5d_950
Frontend logs
fact_main_frontend.log
Not sure if this is related?
[2024-03-17 14:35:32][app][ERROR]: Exception on /admin/logs [GET]
Traceback (most recent call last):
File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/kyle/.local/lib/python3.10/site-packages/flask_restx/api.py", line 674, in error_router
return original_handler(e)
File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
File "/home/kyle/.local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/home/kyle/FACT_core/src/./web_interface/security/decorator.py", line 13, in decorated_view
return fn(*args, **kwargs)
File "/home/kyle/FACT_core/src/./web_interface/components/miscellaneous_routes.py", line 116, in show_logs
backend_logs = '\n'.join(self.intercom.get_backend_logs())
TypeError: can only join an iterable
Other information
Maybe this is a system configuration issue?
This is a problem that has haunted us for quite a while now. There is a limit on how many files can be opened on a system at the same time and the default value is quite low on some systems. You can find out what the limit is by running ulimit -n
and you can set a new value by running ulimit -n <new_limit>
(but this only applies to the current shell, so you may want to put it in your .bashrc/.zshrc/config.fish/etc.). Depending on how many worker processes you configure, you may need more. I'm not running into issues with a value of 9999 but I don't have that many workers configured.