p5py/p5

Too many open files error when loading many images

dcmoura opened this issue · 3 comments

dcmoura commented

Describe the bug
OS error when attempting to load many images.

Traceback (most recent call last):
  File "/Users/dcm/Code/processing-py-play/play2.py.py", line 61, in <module>
    run(renderer="vispy")
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/p5/sketch/userspace.py", line 216, in run
    app.run()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/_default_app.py", line 60, in run
    return default_app.run()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/application.py", line 160, in run
    return self._backend._vispy_run()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/backends/_glfw.py", line 189, in _vispy_run
    self._vispy_process_events()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/backends/_glfw.py", line 178, in _vispy_process_events
    timer._tick()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/backends/_glfw.py", line 512, in _tick
    self._vispy_timer._timeout()
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/app/timer.py", line 160, in _timeout
    self.events.timeout(
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/util/event.py", line 453, in __call__
    self._invoke_callback(cb, event)
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    _handle_exception(self.ignore_callback_errors,
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/vispy/util/event.py", line 469, in _invoke_callback
    cb(event)
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/p5/sketch/Vispy2DRenderer/base.py", line 100, in on_timer
    self.setup_method()
  File "/Users/dcm/Code/processing-py-play/play2.py.py", line 33, in setup
    imgs = [
  File "/Users/dcm/Code/processing-py-play/play2.py.py", line 34, in <listcomp>
    load_image(str(p))
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/p5/core/image.py", line 257, in load_image
    return p5.renderer.load_image(filename)
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/p5/sketch/Vispy2DRenderer/renderer2d.py", line 574, in load_image
    img = Image.open(filename)
  File "/Users/dcm/.virtualenvs/processing-py/lib/python3.9/site-packages/PIL/Image.py", line 2953, in open
    fp = builtins.open(filename, "rb")
OSError: [Errno 24] Too many open files: 'myfile.jpg'

To Reproduce

imgs_paths = Path("/my/images/directory").glob("*.jpg")
imgs = [load_image(str(p)) for p in imgs_paths]

Expected behavior
Expected that load image would close the handle to the file after loading the image

System information:

  • p5 release (version number or latest commit): 0.8.2
  • Python version: 3.9.12
  • Operating system: macOS ventura 13.1

Additional context
I am trying to load hundreds of images

github-actions commented

Thank you for submitting your first issue to p5py

saad-24 commented

It is happening because your OS has a limit of how many files it can open at once. That limit is getting exceeded and that's what bringing the error message. There is one thing you can do:

  1. find out how many files your OS can open at once.
  2. Then go the code where images are opening.
  3. set a condition using if the number of images = OS file limit, then break or close those images.
  4. You can also write code to close an image after it's opened so than this error should not be there anymore.
dcmoura commented

Thank you @saad-24.
I fixed by forcing PIL to load the image, which releases the file handle (i.e. closes the file). That means no lazy loading, but that's ok in my case.

def force_load_image(img_path):
    im = load_image(str(img_path))
    im._img.load()
    return im

imgs_paths = Path("/my/images/directory").glob("*.jpg")
imgs = [force_load_image(p) for p in imgs_paths]

BTW, it's great having Processing on Python :-D
Congrats!

  • 👍1