Dependency resolution takes awfully long time

[maksbotan]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Big Sur

• Poetry version: 1.1.13

• Link of a Gist with the contents of your pyproject.toml file: I can provide upon request.

Issue

Version resolution in our moderately large (25 immediate dependencies) project is taking too long. For me it is about 200-250 seconds each time, and my colleagues have reported times even up to 1000 seconds.

A common scenario in this project is to update version constraint on one of the dependencies or add a new one, and then run poetry lock --no-update. It seems that the algorithm behind this command is not optimal in our case.

I will try to describe what I see in verbose (-vvv) output. Full output may be found here: https://gist.github.com/maksbotan/a72f5d801e8515b74a9f0ab7fbcef321

This concrete run had no changes in pyproject.toml and took 220 seconds anyway. As far as I see, poetry tries different solutions with different version of numpy, even if numpy version is not changed:

   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.18.5) with markers platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", numpy (>=1.19.2) with markers platform_machine == "aarch64" and python_version < "3.10" and numpy (>=1.20.0) with markers platform_machine == "arm64" and python_version < "3.10".
   1: Version solving took 19.610 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}).
...
   1: Version solving took 72.396 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}).
...
   1: Version solving took 67.490 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}}).
   1: Version solving took 60.766 seconds.
   1: Tried 1 solutions.
   0: Complete version solving took 220.588 seconds with 3 overrides
   0: Resolved with overrides: ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.18.5)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.19.2)>}}), ({Package('pandas', '1.4.0'): {'numpy': <Dependency numpy (>=1.20.0)>}})

I do not understand why poetry needs to retry different numpy constraints, especially when nothing changed in that regard, and why it comes up with three overrides for it with intervals that can be simplified to one. For the record, resolved version of numpy is 1.21.4.

Also, this may be related: looks like poetry tries to query global PyPI for information about packages from our private repo, even though they have [package.source] section in poetry.lock. From log:

PyPI: Getting info for humanizator (0.4.9) from PyPI
PyPI: Getting info for humanizator (0.4.9) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI
PyPI: Getting info for oneq (1.11.2) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI
PyPI: Getting info for humanizator (0.4.9) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI
PyPI: Getting info for oneq (1.11.2) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI
PyPI: Getting info for humanizator (0.4.9) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI
PyPI: Getting info for oneq (1.11.2) from PyPI
PyPI: Getting info for pandorabox (0.4.21) from PyPI

Thanks in advance!

[alecandido]

It might be related #4924 (if not the exact same)

[eekcoopuw]

Also looks related to #5121. Note the "retries" name various combinations of versions of pandas and numpy.

[maksbotan]

Indeed, looks like it's the same issue.

I'll leave this open because of the second part though, I think poetry should not try querying PyPI for private packages.

[john-sandall]

For those hitting this, I added a couple tips on this issue on how to improve issues with Poetry being slow: #2094 (comment)

[neersighted]

Closing as duplicate of #5121 and #6713.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.