duo-labs/cloudmapper

Cloudmapper fails to install on macos

serialp opened this issue · 18 comments

serialp commented

Hello, whatever the solution I came up with to setup cloudmapper, it fails at the pyjq install...

pip3 install --user pipenv
cd cloudmapper/; pipenv install --skip-lock

is it related to the below issue ?

doloopwhile/pyjq#54
https://bugs.python.org/issue42504

Please find hereunder my logs:

Installing dependencies from Pipfile...
An error occurred while installing pyjq==2.3.1! Will try again.
An error occurred while installing seaborn==0.10.1! Will try again.
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 37/37 — 00:00:44
Installing initially failed dependencies...
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/cli/command.py", line 233, in install
[InstallError]: retcode = do_install(
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 2052, in do_install
[InstallError]: do_init(
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]: do_install_dependencies(
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]: batch_install(
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]: _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]: raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Collecting pyjq==2.3.1
[pipenv.exceptions.InstallError]: Using cached pyjq-2.3.1.tar.gz (2.1 MB)
[pipenv.exceptions.InstallError]: Requirement already satisfied: six in /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages (from pyjq==2.3.1->-r /var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pipenv-8bez_ooa-requirements/pipenv-r8czubvu-requirement.txt (line 1)) (1.14.0)
[pipenv.exceptions.InstallError]: Building wheels for collected packages: pyjq
[pipenv.exceptions.InstallError]: Building wheel for pyjq (setup.py): started
[pipenv.exceptions.InstallError]: Building wheel for pyjq (setup.py): finished with status 'error'
[pipenv.exceptions.InstallError]: ERROR: Command errored out with exit status 1:
[pipenv.exceptions.InstallError]: command: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-wheel-dbvkaevy
[pipenv.exceptions.InstallError]: cwd: /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/
[pipenv.exceptions.InstallError]: Complete output (44 lines):
[pipenv.exceptions.InstallError]: running bdist_wheel
[pipenv.exceptions.InstallError]: running build
[pipenv.exceptions.InstallError]: running build_py
[pipenv.exceptions.InstallError]: creating build
[pipenv.exceptions.InstallError]: creating build/lib.macosx-11-x86_64-3.8
[pipenv.exceptions.InstallError]: copying pyjq.py -> build/lib.macosx-11-x86_64-3.8
[pipenv.exceptions.InstallError]: warning: build_py: byte-compiling is disabled, skipping.
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: running build_ext
[pipenv.exceptions.InstallError]: Traceback (most recent call last):
[pipenv.exceptions.InstallError]: File "", line 1, in
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 93, in
[pipenv.exceptions.InstallError]: setup(
[pipenv.exceptions.InstallError]: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup
[pipenv.exceptions.InstallError]: return distutils.core.setup(**attrs)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
[pipenv.exceptions.InstallError]: dist.run_commands()
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
[pipenv.exceptions.InstallError]: self.run_command(cmd)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
[pipenv.exceptions.InstallError]: self.run_command('build')
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
[pipenv.exceptions.InstallError]: self.distribution.run_command(command)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
[pipenv.exceptions.InstallError]: self.run_command(cmd_name)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
[pipenv.exceptions.InstallError]: self.distribution.run_command(command)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 30, in run
[pipenv.exceptions.InstallError]: self._build_oniguruma()
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 39, in _build_oniguruma
[pipenv.exceptions.InstallError]: self._build_lib(
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 67, in _build_lib
[pipenv.exceptions.InstallError]: os.environ['MACOSX_DEPLOYMENT_TARGET'] = macosx_deployment_target
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 680, in setitem
[pipenv.exceptions.InstallError]: value = self.encodevalue(value)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 750, in encode
[pipenv.exceptions.InstallError]: raise TypeError("str expected, not %s" % type(value).name)
[pipenv.exceptions.InstallError]: TypeError: str expected, not int
[pipenv.exceptions.InstallError]: ----------------------------------------
[pipenv.exceptions.InstallError]: ERROR: Failed building wheel for pyjq
[pipenv.exceptions.InstallError]: Running setup.py clean for pyjq
[pipenv.exceptions.InstallError]: Failed to build pyjq
[pipenv.exceptions.InstallError]: Installing collected packages: pyjq
[pipenv.exceptions.InstallError]: Running setup.py install for pyjq: started
[pipenv.exceptions.InstallError]: Running setup.py install for pyjq: finished with status 'error'
[pipenv.exceptions.InstallError]: ERROR: Command errored out with exit status 1:
[pipenv.exceptions.InstallError]: command: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-record-1eezjp50/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/include/site/python3.8/pyjq
[pipenv.exceptions.InstallError]: cwd: /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/
[pipenv.exceptions.InstallError]: Complete output (46 lines):
[pipenv.exceptions.InstallError]: running install
[pipenv.exceptions.InstallError]: running build
[pipenv.exceptions.InstallError]: running build_py
[pipenv.exceptions.InstallError]: creating build
[pipenv.exceptions.InstallError]: creating build/lib.macosx-11-x86_64-3.8
[pipenv.exceptions.InstallError]: copying pyjq.py -> build/lib.macosx-11-x86_64-3.8
[pipenv.exceptions.InstallError]: warning: build_py: byte-compiling is disabled, skipping.
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: running build_ext
[pipenv.exceptions.InstallError]: Traceback (most recent call last):
[pipenv.exceptions.InstallError]: File "", line 1, in
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 93, in
[pipenv.exceptions.InstallError]: setup(
[pipenv.exceptions.InstallError]: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup
[pipenv.exceptions.InstallError]: return distutils.core.setup(**attrs)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
[pipenv.exceptions.InstallError]: dist.run_commands()
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
[pipenv.exceptions.InstallError]: self.run_command(cmd)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
[pipenv.exceptions.InstallError]: return orig.install.run(self)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/install.py", line 545, in run
[pipenv.exceptions.InstallError]: self.run_command('build')
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
[pipenv.exceptions.InstallError]: self.distribution.run_command(command)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
[pipenv.exceptions.InstallError]: self.run_command(cmd_name)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
[pipenv.exceptions.InstallError]: self.distribution.run_command(command)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
[pipenv.exceptions.InstallError]: cmd_obj.run()
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 30, in run
[pipenv.exceptions.InstallError]: self._build_oniguruma()
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 39, in _build_oniguruma
[pipenv.exceptions.InstallError]: self._build_lib(
[pipenv.exceptions.InstallError]: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 67, in _build_lib
[pipenv.exceptions.InstallError]: os.environ['MACOSX_DEPLOYMENT_TARGET'] = macosx_deployment_target
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 680, in setitem
[pipenv.exceptions.InstallError]: value = self.encodevalue(value)
[pipenv.exceptions.InstallError]: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 750, in encode
[pipenv.exceptions.InstallError]: raise TypeError("str expected, not %s" % type(value).name)
[pipenv.exceptions.InstallError]: TypeError: str expected, not int
[pipenv.exceptions.InstallError]: ----------------------------------------
[pipenv.exceptions.InstallError]: ERROR: Command errored out with exit status 1: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-record-1eezjp50/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/include/site/python3.8/pyjq Check the logs for full command output.
ERROR: Couldn't install package: pyjq
Package installation failed...
☤ ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/2 —

Many thanks !

w0rmr1d3r commented

It happened the same to me (same error), I tried to install the requirements using:
pip install -r requirements.txt
and with:
pip3 install -r requirements.txt
And it fails with the very same error.

Could it be due to higher versions of Python than 3.7 ? I was using 3.9 and @serialp here was using 3.8.

0xdabbad00 commented

Is this on macOS Big Sur? Could be this Python issue: https://bugs.python.org/issue42504

I installed CloudMapper without issues on an Ubuntu system last week.

  • 👍1
w0rmr1d3r commented

Is this on macOS Big Sur? Could be this Python issue: https://bugs.python.org/issue42504

I installed CloudMapper without issues on an Ubuntu system last week.

Yes, in my case it is. macOS Big Sur 11.1
I'll wait for that to get fixed! Thanks @0xdabbad00 ! Good catch!

serialp commented

@w0rmr1d3r I used the same install method as you then switch to pipenv I also tried with each version of python util 3.9.
That's why I have mentioned this issue to @0xdabbad00 and yes Scott I am using MacOs big sur 11.2.2 which is why I mentioned the related issue if it can help.
For linux have you tried on Ubuntu 20.04 LTS ?
So will you able to deliver a fix on your side or pyjq will handle it ?

Thanks again for your feedback !

w0rmr1d3r commented

@w0rmr1d3r I used the same install method as you then switch to pipenv I also tried with each version of python util 3.9.
That's why I have mentioned this issue to @0xdabbad00 and yes Scott I am using MacOs big sur 11.2.2 which is why I mentioned the related issue if it can help.
For linux have you tried on Ubuntu 20.04 LTS ?
So will you able to deliver a fix on your side or pyjq will handle it ?

Thanks again for your feedback !

Sorry @serialp , I went too fast on your logs and didn't see the links to the very same issue @0xdabbad00 said.
I belive that, regarding to Python issues, we might need to wait for pyjq to handle it.

w0rmr1d3r commented

@serialp the python issue has been resolved! It should work now, have you tried it again?

  • 👍1
serialp commented

@w0rmr1d3r Many thank, but finally went to the docker version, which works fine. We'll try to install it on my MacOs next time i need it !

  • 👍1
cvdv87 commented

I had the same issue on mac os, removing the version numbers from the file fixed it for me, pip sorted out everything.

  • 👀1
w0rmr1d3r commented

@cvdv87 which file? The requirements.txt file?

cvdv87 commented

@w0rmr1d3r correct, my file looks like this:
astroid autoflake autopep8 boto3 botocore certifi chardet coverage docutils idna isort Jinja2 jmespath lazy-object-proxy MarkupSafe matplotlib mccabe mock netaddr nose pandas parliament policyuniverse pycodestyle pyflakes pyjq pylint python-dateutil PyYAML requests s3transfer seaborn six toml typed-ast urllib3 wrapt

I have to say, I had another dependency issue, different packages, but perhaps this works for you too.

  • 🚀1
w0rmr1d3r commented

There are a lot of issues in this repo regarding versions and some incompatibilities (and some PRs opened as well).
We should take a look at those too!
Question:
If we delete fixed versions, are we installing latest versions for all of those dependencies? Does Cloudmapper work as usual?

Which dependency issue you say you were having?

cvdv87 commented

So here is my approach, keep in mind I use a newer python version as advised in the installation page:
Python version 3.8

1)- Clone repo, install pre-reqs, start venv
2)- Update pip: pip install --upgrade pip
3)- pip install -r requirements.txt

Error:
ERROR: Cannot install -r requirements.txt (line 5) and urllib3==1.26.5 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested urllib3==1.26.5
botocore 1.17.6 depends on urllib3<1.26 and >=1.20; python_version != "3.4"

Solution provided by pip:
To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

Solution 2 is what I've tested and afaik everything seems to be working fine for me.

w0rmr1d3r commented

That happens to you the same way it happened to other people, because urllib3 got updated and it conflicted with botocore versions and other dependencies.
Exists this PR which fixes the issue that you have encountered -> #876
Which, at the same time, it fixes this issue -> #875

That's why I was saying that there are issues and PRs opened regarding dependencies and problems installing them.

  • 👍2
Sean12697 commented

Sounds similar to #301 (I've just opened #881, based on Debian)

w0rmr1d3r commented

I can see in #881 that you installed dependencies without a fixed version, which is similar to other PRs that basically upgrade dependencies, am I wrong?

Thanks @Sean12697 !

Sean12697 commented

You're not wrong @w0rmr1d3r, I did manually implement #876 (which I see you reviewed a couple of weeks before I opened my issue), although it was running sudo apt install autoconf libtool from doloopwhile/pyjq#50 which finally fixed the issue for me.

Ideally I would reset my environment to test how much of an impact manually implementing #876 has before installed the additional apt packages.

  • 👍1
  • 🚀1
ishtylerc commented

@w0rmr1d3r correct, my file looks like this: astroid autoflake autopep8 boto3 botocore certifi chardet coverage docutils idna isort Jinja2 jmespath lazy-object-proxy MarkupSafe matplotlib mccabe mock netaddr nose pandas parliament policyuniverse pycodestyle pyflakes pyjq pylint python-dateutil PyYAML requests s3transfer seaborn six toml typed-ast urllib3 wrapt

I have to say, I had another dependency issue, different packages, but perhaps this works for you too.

This is what fixed my issue, removing the verision numbers

danieltanfh95 commented

issues fixed by updating typed-ast==1.5.5 and pyjq==2.6.0

  • 👍1