[Errno 24] Too many open files
joshuaswilcox opened this issue · 17 comments
I just started getting this error, it stops on the same repo each time, but if i cd into that repo and run gitup . it works fine
Fetching origin: up to date.
Updating master: Traceback (most recent call last):
File "/usr/local/bin/gitup", line 9, in <module>
load_entry_point('gitup==0.2.3', 'console_scripts', 'gitup')()
File "build/bdist.macosx-10.10-x86_64/egg/gitup/script.py", line 90, in run
File "build/bdist.macosx-10.10-x86_64/egg/gitup/script.py", line 82, in main
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 296, in update_directories
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 275, in _update_directory
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 256, in _update_subdirectories
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 240, in _update_repository
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 198, in _update_branches
File "build/bdist.macosx-10.10-x86_64/egg/gitup/update.py", line 178, in _update_branch
File "/usr/local/lib/python2.7/site-packages/GitPython-1.0.0-py2.7.egg/git/refs/symbolic.py", line 183, in _get_commit
obj = self._get_object()
File "/usr/local/lib/python2.7/site-packages/GitPython-1.0.0-py2.7.egg/git/refs/symbolic.py", line 176, in _get_object
return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
File "/usr/local/lib/python2.7/site-packages/GitPython-1.0.0-py2.7.egg/git/objects/base.py", line 65, in new_from_sha
oinfo = repo.odb.info(sha1)
File "/usr/local/lib/python2.7/site-packages/gitdb-0.6.4-py2.7-macosx-10.10-x86_64.egg/gitdb/db/base.py", line 205, in info
return self._db_query(sha).info(sha)
File "/usr/local/lib/python2.7/site-packages/gitdb-0.6.4-py2.7-macosx-10.10-x86_64.egg/gitdb/db/loose.py", line 157, in info
m = self._map_loose_object(sha)
File "/usr/local/lib/python2.7/site-packages/gitdb-0.6.4-py2.7-macosx-10.10-x86_64.egg/gitdb/db/loose.py", line 135, in _map_loose_object
return file_contents_ro_filepath(db_path, flags=self._fd_open_flags)
File "/usr/local/lib/python2.7/site-packages/gitdb-0.6.4-py2.7-macosx-10.10-x86_64.egg/gitdb/util.py", line 190, in file_contents_ro_filepath
return file_contents_ro(fd, stream, allow_mmap)
File "/usr/local/lib/python2.7/site-packages/gitdb-0.6.4-py2.7-macosx-10.10-x86_64.egg/gitdb/util.py", line 163, in file_contents_ro
return mmap.mmap(fd, os.fstat(fd).st_size, access=mmap.ACCESS_READ)
mmap.error: [Errno 24] Too many open files
Bizarre... I've never gotten this before. How many repos does it take for the error to trigger? Smells like a bug in gitdb or GitPython – it's possible I need to let something get GC'd that's being missed, but I need to look more carefully at how gitdb's internals work first.
Looks like GitPython has an update out, so can you check if that changes anything?
I have 46 repositories and it fails at about 32 repos
That's weird... I has the same issue yesterday (with only 17 repos) but today it works fine again...
yeah it doesn't always fail, just most of the time
I'm not able to reproduce this...
Edit: I still can't reproduce it, even when setting the open file limit absurdly low (ulimit -n 16) and having gitup operate on ~50 repos.
I updated git-python to the newest version
GitPython (1.0.1)
and am now running Python 2.7.9 and it seems to be working fine
I still had this issue this morning. I updated GitPyhon and now I get this:
File "/usr/local/bin/gitup", line 9, in <module>
load_entry_point('gitup==0.2.2', 'console_scripts', 'gitup')()
File "build/bdist.macosx-10.10-intel/egg/gitup/script.py", line 90, in run
File "build/bdist.macosx-10.10-intel/egg/gitup/script.py", line 85, in main
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 280, in update_bookmarks
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 267, in _update_directory
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 248, in _update_subdirectories
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 231, in _update_repository
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 90, in _fetch_remotes
File "/Library/Python/2.7/site-packages/git/remote.py", line 651, in fetch
res = self._get_fetch_info_from_stderr(proc, progress or RemoteProgress())
File "/Library/Python/2.7/site-packages/git/remote.py", line 557, in _get_fetch_info_from_stderr
for pline in progress_handler(line):
File "/Library/Python/2.7/site-packages/git/util.py", line 277, in handler
return self._parse_progress_line(line.rstrip())
File "/Library/Python/2.7/site-packages/git/util.py", line 267, in _parse_progress_line
message)
File "build/bdist.macosx-10.10-intel/egg/gitup/update.py", line 44, in update
TypeError: object of type 'float' has no len()
Oh my bad >< I thought I was up to date for some reason...
Having the same issue :(
push:
Fetching originTraceback (most recent call last):
File "/usr/local/Cellar/gitup/0.3/libexec/bin/gitup", line 9, in <module>
load_entry_point('gitup==0.3', 'console_scripts', 'gitup')()
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/script.py", line 100, in run
main()
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/script.py", line 92, in main
update_directories(args.directories_to_update, update_args)
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/update.py", line 220, in update_directories
_update_directory(full_path, update_args)
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/update.py", line 201, in _update_directory
_update_subdirectories(path, update_args)
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/update.py", line 185, in _update_subdirectories
_update_repository(repo, *update_args)
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/update.py", line 166, in _update_repository
_fetch_remotes(remotes)
File "/usr/local/Cellar/gitup/0.3/libexec/lib/python2.7/site-packages/gitup/update.py", line 75, in _fetch_remotes
results = remote.fetch(progress=_ProgressMonitor())
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/remote.py", line 643, in fetch
kwargs = add_progress(kwargs, self.repo.git, progress)
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/remote.py", line 51, in add_progress
v = git.version_info
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/cmd.py", line 462, in version_info
return self._version_info
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/cmd.py", line 439, in __getattr__
return LazyMixin.__getattr__(self, name)
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/gitdb/util.py", line 237, in __getattr__
self._set_cache_(attr)
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/cmd.py", line 445, in _set_cache_
version_numbers = self._call_process('version').split(' ')[2]
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/cmd.py", line 834, in _call_process
return self.execute(make_call(), **_kwargs)
File "/usr/local/Cellar/gitup/0.3/libexec/vendor/lib/python2.7/site-packages/git/cmd.py", line 576, in execute
raise GitCommandNotFound(str(err))
git.exc.GitCommandNotFound: [Errno 24] Too many open files
$ gitup --version
gitup 0.3
@wilrnh I found gitpython-developers/GitPython#60. I'm taking a guess here, but it looks like smmap needs an update to 0.9.0. Can you try pip install -U smmap?
Btw, I'm on OSX, and:
$ python --version
Python 2.7.10
Just tried python 3.4 in virtualenv with smmap 0.9.0 and got the same issue fyi
I can definitely confirm that GitPython/gitdb/smmap is leaking file objects. I'll try to see if there's a way to fix it but I may need to report this upstream.
@wilrnh Can you please see if this still happens on the current master version of https://github.com/gitpython-developers/GitPython? Based on gitpython-developers/GitPython#60 this may be a fix.
I have tried and it seems to fix the problem, thanks!