Python error while installing SHPC

Question

Python error while installing SHPC

Closed this issue a year ago · 5 comments

There is an error message related to PIP when installing shpc


Collecting singularity-hpc==0.1.22
  Downloading singularity_hpc-0.1.22-py3-none-any.whl (128 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.2/128.2 kB 5.0 MB/s eta 0:00:00
Collecting Jinja2
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting jsonschema
  Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting requests
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting ruamel.yaml
  Using cached ruamel.yaml-0.17.32-py3-none-any.whl (112 kB)
Collecting spython>=0.2.0
  Using cached spython-0.3.0-py3-none-any.whl (109 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting attrs>=17.4.0
  Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Using cached pyrsistent-0.19.3-py3-none-any.whl (57 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.0.3-py3-none-any.whl (123 kB)
Collecting ruamel.yaml.clib>=0.2.7
  Using cached ruamel.yaml.clib-0.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (485 kB)
Installing collected packages: spython, urllib3, ruamel.yaml.clib, pyrsistent, MarkupSafe, idna, charset-normalizer, certifi, attrs, ruamel.yaml, requests, jsonschema, Jinja2, singularity-hpc
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.3 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 jsonschema-4.17.3 pyrsistent-0.19.3 requests-2.31.0 ruamel.yaml-0.17.32 ruamel.yaml.clib-0.2.7 singularity-hpc-0.1.22 spython-0.3.0 urllib3-2.0.3
--- Logging error ---
Traceback (most recent call last):
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 177, in emit
    self.console.print(renderable, overflow="ignore", crop=False, style=style)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_vendor/rich/console.py", line 1673, in print
    extend(render(renderable, render_options))
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_vendor/rich/console.py", line 1305, in render
    for render_output in iter_render:
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 134, in __rich_console__
    for line in lines:
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_vendor/rich/segment.py", line 249, in split_lines
    for segment in segments:
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_vendor/rich/console.py", line 1283, in render
    renderable = rich_cast(renderable)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_vendor/rich/protocol.py", line 36, in rich_cast
    renderable = cast_method()
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/self_outdated_check.py", line 130, in __rich__
    pip_cmd = get_best_invocation_for_this_pip()
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/utils/entrypoints.py", line 58, in get_best_invocation_for_this_pip
    if found_executable and os.path.samefile(
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/genericpath.py", line 101, in samefile
    s2 = os.stat(f2)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/bin/pip'
Call stack:
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/bin/pip", line 10, in <module>
    sys.exit(main())
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 70, in main
    return command.main(cmd_args)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 101, in main
    return self._main(args)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    self.handle_pip_version_check(options)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 190, in handle_pip_version_check
    pip_self_version_check(session, options)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/self_outdated_check.py", line 236, in pip_self_version_check
    logger.warning("[present-rich] %s", upgrade_prompt)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/logging/__init__.py", line 1489, in warning
    self._log(WARNING, msg, args, **kwargs)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/py-pip-22.2.2-lp7v3s5mhyfhzbz7g33gttwnbeqbfpy4/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 179, in emit
    self.handleError(record)
Message: '[present-rich] %s'
Arguments: (UpgradePrompt(old='22.2.2', new='23.1.2'),)

Answer 1 · 2023-06-27T07:10:23.000Z

this is the error:

FileNotFoundError: [Errno 2] No such file or directory: '/scratch/pawsey0001/cdipietrantonio/stack-blas/2023.07/software/linux-sles15-zen3/gcc-12.2.0/python-3.10.10-4ne6vu4r5wqcbblfdedprmkqwahx3lkg/bin/pip'

I am surprised it is looking for pip inside the python installation path.

Answer 2 · 2023-06-29T06:36:01.000Z

Yes, @marcodelapierre , this was not happening before. Needs some more investigation but it appears a non-blocking issue for now.

Answer 3 · 2023-06-30T12:51:13.000Z

did some testing on a Nimbus Ubuntu VM, and could reproduce your error.

it seems to me like this is the key:

Arguments: (UpgradePrompt(old='22.2.2', new='23.1.2'),)

these are the versions in the current settings.sh for Setonix:

python_name="python"
python_version="3.10.10" # has to match the version in the Spack env yaml
setuptools_version="65.5.0" # has to match the version in the Spack env yaml
pip_version="22.2.2" # has to match the version in the Spack env yaml

Apparently pip is too old for one of the packages; I fixed the installation by using pip@23.0.

Answer 4 · 2023-07-01T08:56:42.000Z

Thanks Marco very much appreciated! I will update the version of pip and try again.

Answer 5 · 2023-07-06T05:28:59.000Z

This issue has been solved with 12fc4a2 .