poetry publish fails when tool.poetry.source is specified throwing "a repository with name was already added"
Jakobhenningjensen opened this issue · 9 comments
Description
When releasing some code we have a system which step 1 is to run a unittest. If that passes, the code is published to our internal packages such that we can use them in production.
The code for the unittest/publish is
set -e
pip install poetry==1.8.3 keyring==24.0.0 keyrings.google-artifactregistry-auth==1.1.1
poetry install
poetry run pytest ./tests/unit_tests
poetry config repositories.my_repo https://my-package/packages"
poetry publish --build -r my_repo
The issue is that the publish part fails when I want to include pytorch with CUDA, thus I have to specify where to get the CUDA-torch from using the tool.poetry.source block.
The publish part throws the warning/error
A repository with name torch-gpu-linux was already added.
and exists with status 1, making the entire pipeline failing since the package cannot be published.
It seems like the repo specified in tool.poetry.source is being added twice, thus failing on the second one.
Workarounds
Not in the .toml-file.
The way we work around it later on, is to skip the torch-part completely and then install it manually later on when the package is downloaded (which easily can lead to bugs).
Poetry Installation Method
pip
Operating System
It is run in a docker-container
Poetry Version
1.8.3
Poetry Configuration
cache-dir = "/root/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
repositories.dinero.url = "https://europe-west4-python.pkg.dev/test-dineroci-cm/ai-packages/"
repositories.torch-gpu-linux.url = "https://download.pytorch.org/whl/cu118"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /root/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"Python Sysconfig
No response
Example pyproject.toml
The (minimalistic) toml-file is
[tool.poetry]
name = "Foo"
version = "0"
description = ""
authors = ["Me"]
readme = "README.md"
[tool.poetry.dependencies]
python = "~3.11"
sentence-transformers = "^2.7.0"
torch = [
{ version = "^2.3.0", markers = "sys_platform == 'darwin'" }, # macOS, no CUDA
{ version = "^2.3.0+cu118", markers = "sys_platform == 'linux'", source="torch-gpu-linux" } # Linux with CUDA
]
[tool.poetry.dev-dependencies]
pytest = "^7.2.0"
[[tool.poetry.source]]
name = "my_repo"
url = "https://my-package/packages"
priority = "primary"
[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[[tool.poetry.source]]
name = "torch-gpu-linux"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"Poetry Runtime Logs
poetry publish --build -r my_repo
Loading configuration file /root/.config/pypoetry/config.toml
[11:37:55][Step 1/1] Adding repository repo (https://my-package/packages) and setting it as primary
[11:37:55][Step 1/1] Adding repository PyPI (https://pypi.org/simple/) and setting it as supplemental
[11:37:55][Step 1/1] Adding repository torch-gpu-linux (https://download.pytorch.org/whl/cu118) and setting it as explicit
[11:37:55][Step 1/1] Adding repository torch-gpu-linux (https://download.pytorch.org/whl/cu118) and setting it as explicit
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] Stack trace:
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 12 /usr/local/lib/python3.11/site-packages/cleo/application.py:327 in run
[11:37:56][Step 1/1] 325│
[11:37:56][Step 1/1] 326│ try:
[11:37:56][Step 1/1] → 327│ exit_code = self._run(io)
[11:37:56][Step 1/1] 328│ except BrokenPipeError:
[11:37:56][Step 1/1] 329│ # If we are piped to another process, it may close early and send a
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 11 /usr/local/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
[11:37:56][Step 1/1] 188│ self._load_plugins(io)
[11:37:56][Step 1/1] 189│
[11:37:56][Step 1/1] → 190│ exit_code: int = super()._run(io)
[11:37:56][Step 1/1] 191│ return exit_code
[11:37:56][Step 1/1] 192│
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 10 /usr/local/lib/python3.11/site-packages/cleo/application.py:431 in _run
[11:37:56][Step 1/1] 429│ io.input.interactive(interactive)
[11:37:56][Step 1/1] 430│
[11:37:56][Step 1/1] → 431│ exit_code = self._run_command(command, io)
[11:37:56][Step 1/1] 432│ self._running_command = None
[11:37:56][Step 1/1] 433│
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 9 /usr/local/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
[11:37:56][Step 1/1] 471│
[11:37:56][Step 1/1] 472│ if error is not None:
[11:37:56][Step 1/1] → 473│ raise error
[11:37:56][Step 1/1] 474│
[11:37:56][Step 1/1] 475│ return terminate_event.exit_code
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 8 /usr/local/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
[11:37:56][Step 1/1] 455│
[11:37:56][Step 1/1] 456│ if command_event.command_should_run():
[11:37:56][Step 1/1] → 457│ exit_code = command.run(io)
[11:37:56][Step 1/1] 458│ else:
[11:37:56][Step 1/1] 459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 7 /usr/local/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
[11:37:56][Step 1/1] 115│ io.input.validate()
[11:37:56][Step 1/1] 116│
[11:37:56][Step 1/1] → 117│ return self.execute(io) or 0
[11:37:56][Step 1/1] 118│
[11:37:56][Step 1/1] 119│ def merge_application_definition(self, merge_args: bool = True) -> None:
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 6 /usr/local/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
[11:37:56][Step 1/1] 59│
[11:37:56][Step 1/1] 60│ try:
[11:37:56][Step 1/1] → 61│ return self.handle()
[11:37:56][Step 1/1] 62│ except KeyboardInterrupt:
[11:37:56][Step 1/1] 63│ return 1
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 5 /usr/local/lib/python3.11/site-packages/poetry/console/commands/publish.py:52 in handle
[11:37:56][Step 1/1] 50│ from poetry.publishing.publisher import Publisher
[11:37:56][Step 1/1] 51│
[11:37:56][Step 1/1] → 52│ publisher = Publisher(self.poetry, self.io)
[11:37:56][Step 1/1] 53│
[11:37:56][Step 1/1] 54│ # Building package first, if told
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 4 /usr/local/lib/python3.11/site-packages/poetry/console/commands/command.py:23 in poetry
[11:37:56][Step 1/1] 21│ def poetry(self) -> Poetry:
[11:37:56][Step 1/1] 22│ if self._poetry is None:
[11:37:56][Step 1/1] → 23│ return self.get_application().poetry
[11:37:56][Step 1/1] 24│
[11:37:56][Step 1/1] 25│ return self._poetry
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 3 /usr/local/lib/python3.11/site-packages/poetry/console/application.py:129 in poetry
[11:37:56][Step 1/1] 127│ project_path = self._io.input.option("directory")
[11:37:56][Step 1/1] 128│
[11:37:56][Step 1/1] → 129│ self._poetry = Factory().create_poetry(
[11:37:56][Step 1/1] 130│ cwd=project_path,
[11:37:56][Step 1/1] 131│ io=self._io,
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 2 /usr/local/lib/python3.11/site-packages/poetry/factory.py:95 in create_poetry
[11:37:56][Step 1/1] 93│
[11:37:56][Step 1/1] 94│ poetry.set_pool(
[11:37:56][Step 1/1] → 95│ self.create_pool(
[11:37:56][Step 1/1] 96│ config,
[11:37:56][Step 1/1] 97│ poetry.local_config.get("source", []),
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] 1 /usr/local/lib/python3.11/site-packages/poetry/factory.py:172 in create_pool
[11:37:56][Step 1/1] 170│ io.write_line(message)
[11:37:56][Step 1/1] 171│
[11:37:56][Step 1/1] → 172│ pool.add_repository(repository, priority=priority)
[11:37:56][Step 1/1] 173│ if repository.name.lower() == "pypi":
[11:37:56][Step 1/1] 174│ explicit_pypi = True
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] ValueError
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] A repository with name torch-gpu-linux was already added.
[11:37:56][Step 1/1]
[11:37:56][Step 1/1] at /usr/local/lib/python3.11/site-packages/poetry/repositories/repository_pool.py:132 in add_repository
[11:37:56][Step 1/1] 128│ Adds a repository to the pool.
[11:37:56][Step 1/1] 129│ """
[11:37:56][Step 1/1] 130│ repository_name = repository.name.lower()
[11:37:56][Step 1/1] 131│ if self.has_repository(repository_name):
[11:37:56][Step 1/1] → 132│ raise ValueError(
[11:37:56][Step 1/1] 133│ f"A repository with name {repository_name} was already added."
[11:37:56][Step 1/1] 134│ )
[11:37:56][Step 1/1] 135│
[11:37:56][Step 1/1] 136│ if default or secondary:
[11:37:56][Step 1/1] Process exited with code 1does not reproduce - as expected I see an error about connecting to "my_repo" but no sign of the error that you are reporting
$ poetry publish --build -r my_repo
Building Foo (0)
- Building sdist
- Built foo-0.tar.gz
- Building wheel
- Built foo-0-py3-none-any.whl
Publishing Foo (0) to my_repo
- Uploading foo-0-py3-none-any.whl FAILED
Connection Error: We were unable to connect to the repository, ensure the url is correct and can be reached.
please provide a way to reproduce
I don't know how else I should reproduce apart from the code above, since it's the one causing the error.
If I remove the torch-gpu-linux source (and just have torch="2.3.0") it works fine
maybe provide a dockerfile in which the error happens, then we should get past any "works on my computer" differences
(still works on my computer, though!)
Just being curious; if you use the "-vvv" flag for publish, is "pytorch-gpu" only being added once?
For some odd reason, changing the version from "0" to "1" in my .toml file seemed to work. I also added "PyPI" as source for the "darwin" source, thus I don't know which of them did the trick but .. 🤷
perhaps you had previously made changes but forgot to hit "save"...
well anyway: I guess either provide a repro or close this out, please
It looks like it is adding torch-gpu-linux twice because it is configured in the poetry config as a repository and the pyproject.toml as a source. Poetry merges these 2 at run time and that is probably why you are running into this specific issue. Remove it from the poetry config and try again.
It looks like it is adding
torch-gpu-linuxtwice because it is configured in the poetry config as a repository and the pyproject.toml as a source. Poetry merges these 2 at run time and that is probably why you are running into this specific issue. Remove it from the poetry config and try again.
Where is the poetry config? I have only modified the pyproject.toml
I always forget where it is but it doesn't matter, you can modify and see the contents using the 'poetry config' command.
'poetry config --unset repositories.' should do it I think.