rank_strings BrokenPipeError: [Errno 32] Broken pipe on macOS
wesinator opened this issue · 8 comments
macOS 10.14.6
python 3.7.6 homebrew
Traceback (most recent call last):
File "/usr/local/bin/rank_strings", line 8, in <module>
sys.exit(argmain())
File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 138, in argmain
args.scores, args.batch)
File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 27, in main
ranker = joblib.load(os.path.join(modeldir, "ranker.pkl"))
File "/usr/local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 598, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/usr/local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 526, in _unpickle
obj = unpickler.load()
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1088, in load
dispatch[key[0]](self)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1376, in load_global
klass = self.find_class(module, name)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 1426, in find_class
__import__(module, level=0)
File "/usr/local/lib/python3.7/site-packages/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset
File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 33, in <module>
_LIB = _load_lib()
File "/usr/local/lib/python3.7/site-packages/lightgbm/basic.py", line 28, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 442, in LoadLibrary
return self._dlltype(name)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib
Referenced from: /usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
Traceback (most recent call last):
File "/usr/local/bin/flarestrings", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/stringsifter/flarestrings.py", line 29, in main
print(match.group().decode('ascii'))
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipeRequirement already satisfied, skipping upgrade: lightgbm==2.1.2 in /usr/local/lib/python3.7/site-packages (from stringsifter) (2.1.2)
Requirement already satisfied, skipping upgrade: numpy==1.17.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (1.17.1)
Requirement already satisfied, skipping upgrade: scikit-learn==0.21.3 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.21.3)
Requirement already satisfied, skipping upgrade: joblib==0.13.2 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.13.2)
Requirement already satisfied, skipping upgrade: pytest==3.10.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (3.10.1)
Requirement already satisfied, skipping upgrade: fasttext==0.9.1 in /usr/local/lib/python3.7/site-packages (from stringsifter) (0.9.1)
Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.7/site-packages (from lightgbm==2.1.2->stringsifter) (1.3.1)
Requirement already satisfied, skipping upgrade: py>=1.5.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.8.0)
Requirement already satisfied, skipping upgrade: more-itertools>=4.0.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (7.2.0)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.12.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (42.0.2)
Requirement already satisfied, skipping upgrade: atomicwrites>=1.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (1.3.0)
Requirement already satisfied, skipping upgrade: pluggy>=0.7 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (0.12.0)
Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /usr/local/lib/python3.7/site-packages (from pytest==3.10.1->stringsifter) (18.2.0)
Requirement already satisfied, skipping upgrade: pybind11>=2.2 in /usr/local/lib/python3.7/site-packages (from fasttext==0.9.1->stringsifter) (2.3.0)
Requirement already satisfied, skipping upgrade: importlib-metadata>=0.12 in /usr/local/lib/python3.7/site-packages (from pluggy>=0.7->pytest==3.10.1->stringsifter) (0.20)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata>=0.12->pluggy>=0.7->pytest==3.10.1->stringsifter) (0.6.0)
Building wheels for collected packages: stringsifter
Building wheel for stringsifter (setup.py) ... done
Created wheel for stringsifter: filename=stringsifter-0.20191202-py3-none-any.whl size=1932401 sha256=9e0f0c617e547b96f9ef88995e6d5006ae9aa54a8bc16cc16b077f1284f7e832
Stored in directory: /Users/admin/Library/Caches/pip/wheels/3c/89/aa/c3bc7a6c171c52f5d4bea07724f5f2fd9fc4e7ad23b5946a8c
Successfully built stringsifter
Installing collected packages: stringsifter
Attempting uninstall: stringsifter
Found existing installation: stringsifter 0.20190907
Uninstalling stringsifter-0.20190907:
Successfully uninstalled stringsifter-0.20190907
Successfully installed stringsifter-0.20191202
probably same issue #10 is reporting, but I can't tell for sure because there is limited info
@phtully ?
Sorry for the delayed reply (was on vacation), but thanks for using StringSifter, @wesinator! Your stack trace seems to indicate that lightgbm might not be installed properly, or one of it's dependencies is missing. I'd like to help you fix this and change the docs/code to make sure other users can troubleshoot the issue in the future.
Let's start with this, if you open up a python interpreter within that environment, and try to import lightgbm manually, do you get the same OsError or BrokenPipeError in your stack trace?
The OSError looks like there may be a missing library dependency for lightgbm, namely libgomp, an OpenMP support library. Does brew install libomp before pip installing everything work?
For Mac users like yourself, we may need to provide additional instructions or edit the requirements.txt to reflect some of these issues that seem to be related to yours:
- https://pypi.org/project/lightgbm/
- microsoft/LightGBM#1369
- https://stackoverflow.com/a/44967912
- https://github.com/Microsoft/LightGBM/blob/master/docs/Installation-Guide.rst#macos
Let me know if one of the above methods helps fix this?
Sorry for the delayed reply (was on vacation), but thanks for using StringSifter, @wesinator! Your stack trace seems to indicate that lightgbm might not be installed properly, or one of it's dependencies is missing. I'd like to help you fix this and change the docs/code to make sure other users can troubleshoot the issue in the future.
Let's start with this, if you open up a python interpreter within that environment, and try to
import lightgbmmanually, do you get the same OsError or BrokenPipeError in your stack trace?
yes, import lightgbm gave the following error:
OSError: dlopen(/usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib
Referenced from: /usr/local/lib/python3.7/site-packages/lightgbm/lib_lightgbm.so
Reason: image not found
running the following fixed it:
brew install libomp
pip install -U lightgbm
ERROR: stringsifter 0.20191202 has requirement lightgbm==2.1.2, but you'll have lightgbm 2.3.1 which is incompatible.
Installing collected packages: lightgbm
Attempting uninstall: lightgbm
Found existing installation: lightgbm 2.1.2
Uninstalling lightgbm-2.1.2:
Successfully uninstalled lightgbm-2.1.2
Successfully installed lightgbm-2.3.1
works with latest lightgbm despite stringsifter req version
running with | less still shows BrokenPipe on exit, but not a big deal
$ flarestrings file.exe | rank_strings | less
Traceback (most recent call last):
File "/usr/local/bin/rank_strings", line 8, in <module>
sys.exit(argmain())
File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 138, in argmain
args.scores, args.batch)
File "/usr/local/lib/python3.7/site-packages/stringsifter/rank_strings.py", line 53, in main
print("\n".join(cutoff_sorted_strings))
BrokenPipeError: [Errno 32] Broken pipe
Ok, 1 step closer at least. This one looks more like a bug to me actually. Out of curiosity, when you're not using less how many lines do you expect from the rank_strings output on file.exe? We may have to change the way we write to stdout, I'm guessing if the number of strings is too long that this might happen using python's print statement like how we're doing it currently.
Note that this seems to be related to the issue raised here: https://twitter.com/cyb3rops/status/1228738821245820928
Note that this seems to be related to the issue raised here: https://twitter.com/cyb3rops/status/1228738821245820928
yes - see above #11 (comment)
In case it helps anyone, what worked for me was brew install lightgbm (which also installs libomp) then pip uninstall lightgbm && pip install -U lightgbm