Poetry is extremely slow when resolving the dependencies

Question

Poetry is extremely slow when resolving the dependencies

qiuwei opened this issue 5 years ago · 271 comments

qiuwei commented 5 years ago

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Centos 7
Poetry version: 1.0.0
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/qiuwei/a0c7eee89e5e8d75edb477858213c30b

Issue

I created an empty project and run poetry add allennlp. It takes ages to resolve the dependencies.

Answer 1 · 2020-03-04T16:50:19.000Z

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

Answer 2 · 2020-03-06T05:09:59.000Z

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

It seems so. I have checked the detailed log, poetry kept retrying to resolve the dependency for botocore, but without success. So I assume that the dependency can be eventually resolved if enough time is given.

However, is there any way to get around this?

BTW, I also consider it's better to give some warning if there are some dependencies are not properly specified and could not be resolved after a number of attempts.

Answer 3 · 2020-03-06T19:53:49.000Z

Hi,

I encounter a similiar problem on my MacOS. Python version used is 3.7.6, Poetry is 1.0.5. I just created a new project with no dependencies so far in pyproject.toml, just initially pytest. It takes ages until the new virtualenv is setup with all 11 packages installed.

Running it with -vvv does not bring any new findings.

Regards, Thomas

Answer 4 · 2020-03-08T16:37:27.000Z

Yes, i'm running into the same problem. Resolving dependencies takes forever. I tried to use VPN to get through the GFW, nevertheless, it is still not working. I also tried to change pip source and wrote local source in the toml file, neither works. It's driving me nuts.

Answer 5 · 2020-03-09T10:32:11.000Z

same here...😱

Answer 6 · 2020-03-29T11:40:06.000Z

Same here. I just created an empty project then ran poetry install and it takes so much time to resolve dependencies.

Answer 7 · 2020-03-30T00:22:35.000Z

I'm currently using this workaround:

poetry export -f requirements.txt > requirements.txt
python -m pip install -r requirements.txt
poetry install

It takes a lower time space to install the package locally since all deps are already installed.

Make sure to run poetry shell before to access the created virtual environment and install on it instead of on user/global path.

Answer 8 · 2020-04-06T08:22:23.000Z

Poetry being slow to resolve dependencies seems to be a reoccuring issue:

#476 - Poetry resolving dependencies is amazingly slow
#819 - Resolving dependencies are slow
#832 - Poetry update never finishes resolve and Poetry show --outdated hangs
#1047 - Poetry doesn't resolve on MacOs Mojave

Answer 9 · 2020-04-16T03:18:43.000Z

Maybe there is a dependency conflict.

Answer 10 · 2020-04-19T18:23:40.000Z

No conflict. Poetry is slow as hell.

Answer 11 · 2020-04-19T18:39:45.000Z

First of all, I want to say there is ongoing work to improve the dependency resolution.

However, there is so much Poetry can do with the current state of the Python ecosystem. I invite you to read https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow to know a little more about why the dependency resolution can be slow.

If you report that Poetry is slow, we would appreciate a pyproject.toml that reproduces the issue so we can debug what's going on and if it's on Poetry's end or just the expected behavior.

@gagarine Could you provide the pyproject.toml file you are using?

Answer 12 · 2020-04-19T18:50:14.000Z

Take about 2min to resolve dependencies after adding newspaper3k on a fresh project.
Connexion: 40ms ping and 10Mb/s down.

pyproject.toml

[tool.poetry]
name = "datafox"
version = "0.1.0"
description = ""
authors = ["Me <mail@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"
newspaper3k = "^0.2.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Answer 13 · 2020-04-19T20:19:49.000Z

Hey dudes - as Sebastian implied, the root cause is the Python eco's inconsistent/incomplete way of specifying dependencies and package metadata. Unfortunately, the Pypi team is treating this as a wont fix.

In particular, using the Pypi json endpoint, an empty dep list could either mean "no dependencies", or "dependencies not specified". The Pypi team doesn't want to differentiate between these two cases for reasoning I don't follow.

The soln is to workaround this by maintaining a sep cache from Pypi that properly handles this distinction, and perhaps refuse to use packages that don't properly specify deps. However, this latter aspect may be tough, due to long dep nests.

Python's grown a lot over the decades, and it has much remaining from its early days. There's a culture of no-breaking-changes at any cost.

Having to run arbitrary python code to find dependencies is fucked, but .... we can do this for each noncompliant package, and save it.

Answer 14 · 2020-04-19T21:06:45.000Z

First, it's capitalized PyPI.

Second, there is no way for PyPI to know dependencies for all packages without executing arbitrary code -- which is difficult to do safely and expensive (computationally and financially). PyPI is run on donated infrastructure from sponsors, maintained by volunteers and does not have millions of dollars of funding like many other language ecosystems' package indexes.

For anyone interested in further reading, here's an article written by a PyPI admin on this topic: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies/

Answer 15 · 2020-04-19T21:45:15.000Z

It's not as tough as you imply.

You accept some risk by running the arbitrary code, but accepting things as they are isn't the right approach. We're already forcing this on anyone who installs Python packages; it's what triggers the delays cited in this thread.

I have the above repo running on a $10/month Heroku plan, and it works well.

I've made the assumption that if dependencies are specified, they're specified correctly, so only check the ones that show as having no deps. This won't work every time, but does in a large majority of cases.

Related: Projects like Poetry are already taking a swing at preventing this in the future: Specifying deps in pyproject.toml, Pipfile etc.

Answer 16 · 2020-04-20T00:30:36.000Z

A personal Heroku app is not going to be as valuable a target as PyPI would be. Neither is a $10/month Heroku app going to be able to support the millions of API requests that PyPI gets everyday. The problem isn't in writing a script run a setup.py file in a sandbox, but in the logistics and challenges of providing it for the entire ecosystem.

"It works 90% of the time" is not an approach that can be taken by the canonical package index (which has to be used by everyone) but can be taken by specific tools (which users opt into using). Similar to how poetry can use an AST parser for setup.py files which works >90% of the time, to avoid the overhead of a subprocess call, but pip shouldn't.

Anyway, I wanted to call out that "just blame PyPI folks because they don't care/are lazy" is straight up wrong IMO -- there are reasons that things are the way they are. That doesn't mean we shouldn't improve them, but it's important to understand why we're where we are. I'm going to step away now.

Answer 17 · 2020-04-20T01:17:38.000Z

Before you step away - Can you think of a reason PyPi shouldn't differentiate between no dependencies, and missing dependency data?

If going through existing releases is too bold, what about for new ones?

Answer 18 · 2020-04-20T22:21:56.000Z

I'm new to (more serious) python and don't understand the big drama. Yet setup.py seems a powerful and very bad idea. Dependency management is terrible in python, because of setup.py?

Can someone post a couple of examples where a txt file is not enough andsetup.py was absolutely necessary?

"is a feature that has enabled better compatibility across an increasingly broad spectrum of platforms."

Cargo do it like this: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#platform-specific-dependencies this is not enough for python?

Why poetry does not create their own package repository, avoiding setup.py and using their own dependency declaration? Could take time... but a bot can automatise the pull request on most python module based on the kind of technics used in https://github.com/David-OConnor/pydeps

Answer 19 · 2020-04-20T22:55:56.000Z

I think the root cause is Python's been around for a while, and tries to maintain backwards compatibility. I agree - setup.py isn't an elegant way to do things, and a file that declares dependencies and metadata is a better system. The wheel format causes dependencies to be specified in a MANIFEST file, but there are still many older packages that don't use this format.

As a new lang, Rust benefited by learning from the successes and failures of existing ones. Ie it has nice tools like Cargo, docs, clippy, fmt etc. It's possible to to implement tools / defaults like this for Python, but involves a big change, and potentially backwards-incompatibility. There are equivalents for many of these (pyproject.toml, black etc), but they're not officially supported or widely-adopted. Look at how long it took the Python 3 to be widely adopted for a taste of the challenge.

Answer 20 · 2020-04-21T05:57:29.000Z

Can someone post a couple of examples where a txt file is not enough andsetup.py was absolutely necessary?

Not absolutely necessary, but helpful in the following scenario:

A package has extra A and B
Extra B needs extra A

With setup.py, you can follow the DRY principle:

requires_a = ('some', 'thing')
requires_b = requires_a + ('foo', 'bar')

For requirements.txt, I'm on the one hand not sure how you denote extras at all and even if you can, you would need to repeat the requirements of a within the requirements of b. This is prone to human error.

However, while creating the package, the package builder could output a textfile having those requirements.

Answer 21 · 2020-04-21T06:03:37.000Z

Why poetry does not create their own package repository

You mean replacing PyPI? Good luck with that. I analyzed the packages on PyPI in January (PyPI Analysis 2020):

208,492 packages in total
2,957 had a pyproject.toml
1,511 specified poetry as a tool

I also gave a course about packaging in Python this year to PhD students. They simply want to share there work to a broad audience. I only mentioned poetry briefly because it is such a niche right now.

Changing a big, working system is hard. It took Python 2 -> 3 about 12 years and it is still not completely finished.

Answer 22 · 2020-04-21T09:07:01.000Z

Hi,

I would like to invite everone interested in how depedencies should be declared to this discussion on python.org

fin swimmer

Answer 23 · 2020-04-21T10:15:00.000Z

@finswimmer I check the discussion. Seem like they are reinventing the wheel instead of copy/past something that works (composer, Cargo, ...).

For requirements.txt, I'm on the one hand not sure how you denote extras at all and even if you can, you would need to repeat the requirements of a within the requirements of b. This is prone to human error.

For sure requirements.txt is not good.

You mean replacing PyPI? Good luck with that.

Yes. But why making poetry if it's not to replace PyPI and requirements.txt?

If poetry is compatible with PyPI, there is no incentive to add a pyproject.toml. Perhaps I don't even know I should add one. Now if every time I try to install a package that has no pyproject.toml the command line proposes me to open an issue on this project with a ready to use a template, this could speed things up.

Answer 24 · 2020-04-22T18:44:25.000Z

Can you think of a reason PyPi shouldn't differentiate between no dependencies, and missing dependency data?

It'd be more productive to file an issue on https://github.com/pypa/warehouse, to ask this. There's either a good reason, or PyPI would be open to adding this functionality. In the latter case, depending on how the details work out, it might need to be standardized like pyproject.toml was before poetry adopted it, so that the entire ecosystem can depend on and utilize it.

Answer 25 · 2020-04-23T16:07:22.000Z

Yes. But why making poetry if it's not to replace PyPI and requirements.txt?

You seem to confuse multiple parts of the ecosystem. I would distinguish those entities:

The software which people want to share
Software Repository: The platform on which people want to share it (e.g. PyPI)
Package Format: The format in which they want to share it (e.g. wheels)
Package Builder: The software people want to use to build the package (setuptools / poetry)
Package Uploader: The software people want to use to upload it (twine / poetry)
Package Manager: The software people want to use to install the package (pip / poetry) and its dependencies.
Environment Manager: The software people use to encapsulate (pipenv / poetry)

Under the hood, I think, poetry uses a couple of those base tools. It is just meant to show a more consistent interface to the user.

Answer 26 · 2020-04-23T19:43:24.000Z

I realise that now as I mention in #2338 . I'm therefor not that interested in poetry at the moment. I taught it was like composer and https://packagist.org, but it looks mostly like a wrapper around differents legacy tools.

Answer 27 · 2020-04-23T20:42:27.000Z

[poetry] looks mostly like a wrapper around differents legacy tools

That is not the case. All tools I've mentioned are wide-spread, used by a majority of the Python developers and under active development. Yes, some of the tools are old - pip, for example, is 9 years old. Old is not the same as legacy. The hammer is an old tool. And still people use it. Why? Because it does the job it was designed for.

I don't know PHP well enough to be sure, but I think packagist.org is for PHP what pypi.org is for Python. Composer seems to be a package manager and thus comparable to pip. As composer also supports dependency management during project development, it fills a similar niche as poetry does.

Answer 28 · 2020-05-23T07:52:37.000Z

I figured I would add more to this issue. It's taking more than 20 minutes for me:

gcoakes@workstation ~/s/sys-expect (master) [1]> time poetry add --dev 'pytest-asyncio'
The currently activated Python version 3.7.7 is not supported by the project (^3.8).
Trying to find and use a compatible version. 
Using python3.8 (3.8.2)
Using version ^0.12.0 for pytest-asyncio

Updating dependencies
Resolving dependencies... (655.1s)

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing pytest-asyncio (0.12.0)

________________________________________________________
Executed in   20.98 mins   fish           external 
   usr time    4.96 secs    0.00 micros    4.96 secs 
   sys time    0.35 secs  560.00 micros    0.35 secs

This is the pyproject.toml:

[tool.poetry]
name = "sys-expect"
version = "0.1.0"
description = ""
readme = "README.md"
include = [
    "sys_expect/**/*.html",
    "sys_expect/**/*.js",
]

[tool.poetry.dependencies]
python = "^3.8"
pyyaml = "^5.3.1"
serde = "^0.8.0"
aiohttp = "^3.6.2"
async_lru = "^1.0.2"
astunparse = "^1.6.3"
coloredlogs = "^14.0"
aiofiles = "^0.5.0"

[tool.poetry.dev-dependencies]
pytest = "^5.4"
black = "^19.10b0"
isort = { version = "^4.3.21", extras = ["pyproject"] }
flakehell = "^0.3.3"
flake8-bugbear = "^20.1"
flake8-mypy = "^17.8"
flake8-builtins = "^1.5"
coverage = "^5.1"
pytest-asyncio = "^0.12.0"

[tool.poetry.scripts]
sys-expect = 'sys_expect.cli:run'

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
line_length = 88

[tool.flakehell.plugins]
pyflakes = ["+*"]
flake8-bugbear = ["+*"]
flake8-mypy = ["+*"]
flake8-builtins = ["+*"]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Answer 29 · 2020-05-23T10:30:41.000Z

Question: What exactly does poetry does extra here, that makes it way much slower than pip's dependency resolution?

Does it actually put a lot of extra effort to figure out dependencies in a lot of situations that pip doesn't?

Edit: It doesn't

As it is now, pip doesn’t have true dependency resolution, but instead simply uses the first specification it finds for a project.

https://pip.pypa.io/en/stable/user_guide/#requirements-files

Answer 30 · 2020-05-23T12:26:10.000Z

Pip doesn't have dependency resolution.

Answer 31 · 2020-05-23T13:37:27.000Z

@David-OConnor It doesn't? But when I install a package in a blank environment, I usually see a lot of packages installed. Isn't that dependency resolution?

Answer 32 · 2020-05-23T13:45:29.000Z

That's the extent of it - it'll install sub dependencies, but whichever one you install last wins. Tools like Poetry, Cargo, npm, pyflow etc store info about the relationship between all dependencies, and attempt to find a solution that satisfies all constraints. The particular issue of this thread is that the python ecosystem provides no reliable way of determining a packages dependencies without installing the package.

Answer 33 · 2020-05-23T13:54:01.000Z

Cool, thank you for clarifying :-)

the python ecosystem provides no reliable way of determining a packages dependencies without installing the package

When a package is in WHEEL format, I see in the dist directory a METADATA file which contains:

Requires-Dist: click

That seems to be the dependency of the package (see pep-0345). Isn't that a way to get the dependencies without installing the package?

Answer 34 · 2020-05-23T14:00:22.000Z

Anecdotally, I've found that if dependencies are listed listed there, they're accurate. Here are the issues:

Not all packages use wheels (Although you can build wheels, which will generate that info)
This info isn't guaranteed to be accurate.
If this info is absent, there's no way to tell if this means a package has no dependencies, or if the dependencies aren't specified.

My proposal is to use the metadata if it's present, and if not, build the package, determine what the dependencies are and cache, but most people are more conservative, ie pt #2 is a dealbreaker.

We had a discussion higher up in the thread about this, if you'd like more info.

Answer 35 · 2020-05-23T16:55:25.000Z

@David-OConnor, what's your suggestion for resolving things in the immediate term. How can I determine which package is causing the slow down? I am more than happy to make a PR to whichever project that is, but as it is now, any changes to pyproject.toml takes upwards of 20 minutes. When I run with -vvv, I see 1: derived: pyyaml (^5.3.1) as the last line before it hangs for several minutes, but I would assume you are doing installation asynchronously or something.

Answer 36 · 2020-05-23T17:22:49.000Z

@gcoakes I don't have the exact dep graph, but from running it through the package manager I use, there are conflicting requirements for six: a constraint somewhere requires >= 1.13.0, and another requires 1.0.0 exactly. Poetry doesn't install more than one version of a dependency, so it's unsolvable. Not sure why it doesn't just say that though instead of hanging.

Answer 37 · 2020-06-18T21:49:10.000Z

Just to add another data point to the conversation. Running poetry update on many of our projects now takes > 15 minutes.

I understand that doing a comparison between pip and poetry install is not an apples for apples comparison, and also that there are many variables outside poetry's control - however it is hard to believe that 15 minutes resolving a small number of dependencies is unavoidable.

I created a vaguely representative list of dependencies for our projects and put the identical deps in both a pyproject.toml (see https://gist.github.com/jacques-/82b15d76bab3540f98b658c03c9778ea) and Pipfile (see https://gist.github.com/jacques-/293e531e4308bd4d6ad8eabea5299f57).

Poetry resolved this on my machine in around 10-11 minutes, while pipenv did the same in around 1 - 1:15 minutes. This is a roughly 10x improvement.

Unless I'm missing a big part of the puzzle here both pipenv and poetry are doing similar dependency resolution, and are working from the same repositories, so there is no external reason the performance should be this different. Would be great to see prioritising this issue and some of the proposed fixes that are ready to merge e.g. #2149

P.S. thanks for making an awesome tool, poetry has made our lives better since we started using it!

Answer 38 · 2020-06-19T10:06:28.000Z

Maybe a stepping stone to a solution could be to add a flag to show some more info regarding dependency resolution - e.g. for each package, how long it took, and issues encountered/processes used. This would at least let us see where slowdowns are coming from, and potentially let us send PRs to other projects to provide better/more appropriate metadata?

Answer 39 · 2020-07-13T13:55:41.000Z

geopandas seems to take a particularly long time. This was on a new project as the first dependency:

Resolving dependencies... (3335.6s)

Answer 40 · 2020-07-16T08:42:46.000Z

In my case the slow dependency resolution in Poetry was related to an IPv6 issue (also see this related answer on StackOverflow). Temporarily disabling IPv6 solved it. On Ubuntu this can be achieved using the following commands:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Answer 41 · 2020-07-22T16:24:39.000Z

Follwing @lmarsden suggestion I managed to speed up the process by making sure that sites/servers that prefer ipv4 use ipv4. On ubuntu I modified /etc/gai.conf by removing # (uncommenting) the following line:
# precedence ::ffff:0:0/96 100

Answer 42 · 2020-08-06T21:48:27.000Z

I just noticed that the issue for me seemed related to using boto3 without specifying a version in the package I was importing. So I had package A that I built using poetry with boto3 = '*'. That did seem to resolve fairly quickly. But when I tried to import package A into a new package, B, it took >10 minutes to resolve (if it would ever finish). I specified the version used by package A for boto3 in package B, and it resolved my dependencies in < 30 seconds.

Answer 43 · 2020-08-16T14:09:46.000Z

Maybe a stepping stone to a solution could be to add a flag to show some more info regarding dependency resolution - e.g. for each package, how long it took, and issues encountered/processes used. This would at least let us see where slowdowns are coming from, and potentially let us send PRs to other projects to provide better/more appropriate metadata?

This is a pragmatic approach to the problem. Are there any logging / debugging flags we can enable to show installation metadata?

Answer 44 · 2020-10-10T15:54:03.000Z

for any one who come from china mainland (如果你来自**大陆), add this to pyproject.toml

[[tool.poetry.source]]
name = "aliyun"
url = "https://mirrors.aliyun.com/pypi/simple/"
default = true

Answer 45 · 2020-10-10T19:14:58.000Z

for any one who come from china mainland (如果你来自**大陆), add this to pyproject.toml
[[tool.poetry.source]]
name = "aliyun"
url = "https://mirrors.aliyun.com/pypi/simple/"
default = true

Why?

Answer 46 · 2020-10-11T00:43:34.000Z

for any one who come from china mainland (如果你来自**大陆), add this to pyproject.toml
[[tool.poetry.source]]
name = "aliyun"
url = "https://mirrors.aliyun.com/pypi/simple/"
default = true
Why?

A pypi mirror with faster network access in mainland China.

Answer 47 · 2020-10-13T06:51:16.000Z

Well, "slow" is an understatement, I left poetry update running overnight and it's still going at 100% CPU and using 10.04 GB of memory:

Answer 48 · 2020-10-13T08:45:26.000Z

@intgr is there a pyproject.toml you can share that reproduces this behavior?

Answer 49 · 2020-10-13T09:36:37.000Z

@abn Sure, I'll submit a new issue soon. I managed to get it working, poetry update now runs in 22 seconds.

Answer 50 · 2020-10-19T18:50:22.000Z

Another example. Adding black took 284 seconds.

% poetry add --dev black
Using version ^20.8b1 for black

Updating dependencies
Resolving dependencies... (284.1s)

Writing lock file

Package operations: 6 installs, 0 updates, 0 removals

  • Installing appdirs (1.4.4)
  • Installing mypy-extensions (0.4.3)
  • Installing pathspec (0.8.0)
  • Installing typed-ast (1.4.1)
  • Installing typing-extensions (3.7.4.3)
  • Installing black (20.8b1)

Unfortunately I can't share the pyproject.toml.

Answer 51 · 2020-10-26T21:34:00.000Z

I also experienced the same problem on a fresh debian 10.6 install. Funny enough everything runs fine on my linux-mint laptop (same poetry version 1.1.4). Also pip itself seemed to be very slow.

Both pip and poetry started to being usable again once I disabled IPv6 with:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

This disables IPv6 only temporarily until the next reboot. Obviously this only treats a symptom and not the root cause. Hopefully others can chip in and trace the breadcrumbs ...

See https://stackoverflow.com/questions/50787522/extremely-slow-pip-installs

Answer 52 · 2020-10-27T18:01:09.000Z

Discussions here are quite interesting for a noob like me. I have a very naïve question though. Are packages built/published with poetry "correctly specifying their dependencies". In other words, imagine I only add packages built with poetry, will the resolving phase be lightening fast? Or will this still apply:

... the python ecosystem provides no reliable way of determining a packages dependencies without installing the package

I first came here because I though 40s for resolving a single package dependencies was slow, but when I see minutes and hours to the counter I suppose it is normal.

I guess it's not a good idea to use poetry for creating docker images (in CI pipelines for example)?

Answer 53 · 2020-10-27T18:07:56.000Z

Packages built using Poetry (or wheel in general) presumably provides the required information to Pypi, but Poetry doesn't use this; it treats all packages as if they must be installed to verify dependencies.

Answer 54 · 2020-10-27T18:34:26.000Z

As far as I know:
If your project and all its dependencies (and their dependencies) are available for your platform (Python interpreter minor version, operating system, and CPU bitness) as wheels, then it is the best case scenario. Because in a wheel, the dependencies are defined statically (no need to build a sdist to figure out what the exact dependencies are, for example).

You, as the developer (maintainer) of a project, the best you can do to help lower the difficulty of dependency resolution for everyone else, is to distribute the wheels of your project for as many platforms as possible (upload the .whl files to PyPI). Often projects (libraries, applications) are made of pure Python code (no C extension for example) so just 1 wheel is enough to cover all platforms.

Answer 55 · 2020-10-27T18:34:28.000Z

@David-OConnor Is there a technical reason for that? Isn't it possible to check wether a package correctly specifies its dependencies?

Answer 56 · 2020-10-27T18:37:29.000Z

Is there a technical reason for that?

@cglacet What do you mean?

Answer 57 · 2020-10-27T18:38:55.000Z

@MartinThoma I was asking why "Poetry doesn't use this (information)".

Answer 58 · 2020-10-27T18:49:53.000Z

Ultimately, the issue lies with PyPi having no flag to distinguish properly-specified (ie wheel) vs improperly. Poetry accepts this, instead of attempting to work around it.

Answer 59 · 2020-10-27T19:26:09.000Z

@cglacet I think what @David-OConnor (please correct me if I'm wrong) might be referring to, is the difference between source distributions (sdist) and pre-built distributions (such as wheels).

In short and simplified:
On PyPI there are 2 types of distribution formats: sdist and wheel. For our concern here (dependency resolution), the meaningful difference between these 2 formats is that the info contained in sdist is not reliable (for various reasons, legitimate ones and some much less so). Now when poetry is doing dependency resolution for project App that depends on Lib, and it so happens that Lib is only available as sdist, poetry needs to build that sdist locally (which can take a large amount of time), to figure out if Lib has dependencies of its own and what they really are. If Lib were to be available as wheel, then it would be much easier for poetry to figure out Lib's dependencies, because the wheel format specifies that all meta-information it contains is static and entirely reliable, and as a consequence no build step is necessary for the dependency resolution of wheels.

Answer 60 · 2020-10-27T19:35:46.000Z

Yep. Anecdotally, if deps are specified at all on PyPi, they're probably accurate. If not, it could mean either deps aren't specified, or there are no deps.

Pypi not fixing this is irresponsible. Yes, I'm throwing spears at people doing work in their free time, but this needs to be fixed, even if that means being mean.

Answer 61 · 2020-10-27T19:44:02.000Z

@sinoroc @David-OConnor So if I get it correctly there are two problems, 1) we need to retrieve the whole package to answer any question about it, 2) answering dependency questions could potentially take some time because you need to open some files. I can very well understand why 1) is a problem because package registry are slow, but why is 2) really a problem?

Ins't there a cache for storing/reading already installed packages? I mean, when I run pip install on my machine (or even on gitlab) it rarely download the package (because I already have it stored in cache). So unless I run some package updates this should only sum up to problem 2 (for which you can build a cache too package name -> dependencies).

Are there some references I could use to read more about this without adding interferences to this thread (I find it interesting but that's probably off-topic for maintainers or even most users).

Thanks again for your time.

Answer 62 · 2020-10-27T19:51:55.000Z

@David-OConnor

if deps are specified at all on PyPi, they're probably accurate. If not, it could mean either deps aren't specified, or there are no deps.

I do not understand what you mean. Dependencies are not specified in PyPI, they are specified in the distributions.

PyPI does have some insight into what is contained in those distributions. But if that insight is based on the unreliable info contained in the sdist then it is unreliable insight.

Short of building wheels of all existing sdist for all existing combinations of 1. Python interpreter minor version, 2. operating system, 3. CPU bitness, there is no way PyPI can deliver reliable information for the dependency resolver.

Pypi not fixing this is irresponsible. Yes, I'm throwing spears at people doing work in their free time, but this needs to be fixed, even if that means being mean.

The fix is here: use wheels! Anyone (you) can go help any project to help them build wheels and upload them to PyPI. I do not think this is PyPI's role to intervene here. What other solution do you have in mind?

Answer 63 · 2020-10-27T20:05:17.000Z

@cglacet

So if I get it correctly there are two problems, 1) we need to retrieve the whole package to answer any question about it, 2) answering dependency questions could potentially take some time because you need to open some files. I can very well understand why 1) is a problem because package registry are slow, but why is 2) really a problem?

1. Yes. True for both wheels and sdists. They have to be downloaded. Although there is some ongoing work that would result in the possibility to skip the download for the wheels.

2. Yes and no. True for both wheels and sdists, these archives have to be "opened" and some files have to be read to figure if there are dependencies and what they are. But this is not the part that is slow. The slow part, is that for sdists (not for wheels) just opening the archive and reading some files is not enough, those files have to be built (execute the setup.py for example) and in some cases a resource intensive compilation step is necessary (C extensions for example need to be compiled with a C compiler which is usually the very slow bit of the whole process).

Ins't there a cache for storing/reading already installed packages?

As far as I know, there is and subsequent dependency resolutions for the same projects should be faster (download and build steps can be partially be skipped). The wheels built locally in previous attempts are reused.

Are there some references I could use to read more about this without adding interferences to this thread (I find it interesting but that's probably off-topic for maintainers or even most users).

Yes, a bit off-topic but I believe it is helpful for the next users wondering the slow dependency resolution to read some insight into why.

Some good reading I could find on the spot:

https://packaging.python.org/guides/distributing-packages-using-setuptools/#packaging-your-project

Update:

Thinking about it more, I realize I might have mischaracterised things. Getting the metadata (including dependency requirements) out of an sdist, does not require compiling the C extensions (setup.py build). It should be enough to get the egg info (setup.py egg_info).

Answer 64 · 2020-10-27T20:07:07.000Z

Ref https://pypi.org/pypi/requests/json info -> requires_dist

There is no reason a package manager needs to download, install, and examine each package each user installers.

I've already implemented a solution, which is the cache and dependency manager I've posted earlier. See how package managers in other languages like Rust handle this. It builds packages/examines dependencies once total for each package/version, then stores it online. Even if all packages switch to wheels, the problem won't be solved until we have a queryable database of the dependencies... ie PyPi.

Answer 65 · 2020-10-27T20:23:14.000Z

Ref https://pypi.org/pypi/requests/json info -> requires_dist

There is no reason a package manager needs to download, install, and examine each package each user installers.

Yes. I forgot about that. I think poetry uses PyPI's JSON API. pip doesn't. But the same story again, this info is only available for pre-built distributions (wheels), and not for sdists. Those still need currently to be downloaded and built locally (not installed).

I've already implemented a solution, which is the cache I've posted earlier.

You mean your pydeps project? I should look into that, I do not know it yet. There is for sure some room for improvement on PyPI's side, I do not think anyone would deny that, but it is a slow process.

As far as I know, PyPI would like to stay out of the business of building sdists. Maybe some 3rd party organisation would be willing to do that work and deliver the results to PyPI. There are platforms such as libraries.io who could be good candidates for such work. They already built a DB of dependencies (no idea how reliable it is).

Even if all packages switch to wheels, the problem won't be solved until we have a queryable database of the dependencies... ie PyPi.

Well the database is "queryable" via the JSON API as you have shown. So that is already done. Isn't it?

[I am writing this of the top of my head according to the bits of info I have gathered here and there along the way. I do not have insight into all the processes involved, so there might be some inaccuracies. Feel free to correct me.]

Answer 66 · 2020-10-27T20:27:23.000Z

Yep, I was referring to Pydeps.

You brought up a point I hadn't considered: Different dependencies depending on OS etc. That sounds like an important consideration I haven't looked in to. Is this common, in your experience? Ie different dep sets for a single package/version.

The database on the JSON API queryable, but the info is only useful if there's at least 1 dependency listed. Otherwise, it might be that there are no deps. (Which is good info), or simply that they're not specified, ie due not not having a wheel. A simple fix would be for the PyPi requires_dist field to return different things for these cases. Perhaps an empty list if no deps, null etc if not specified. That alone would let Poetry etc reduce the cases it has to download and build for.

Answer 67 · 2020-10-27T20:44:53.000Z

Different dependencies depending on OS etc. That sounds like an important consideration I haven't looked in to. Is this common, in your experience? Ie different dep sets for a single package/version.

Yes, it is problematic. I think there are some setup.py files that look like this (pseudo code):

dependencies = []
if sys.platform == 'linux':
    dependencies.append('This')
elif sys.platform == 'windows':
   dependencies.append('That')
setuptools.setup(install_requires=dependencies)

That makes an sdist completely unreliable. It has to be built on the target platform to know what the dependencies truly are. The correct way is (pseudo-code):

DEPENDENCIES = [
    'This; sys_platform == "linux"',
    'That; sys_platform == "windows"',
]
setuptools.setup(install_requires=DEPENDENCIES)

which is much more reliable. Still not 100% reliable, but that would be somewhat workable. Truth is: as long as there is an executable setup.py file in the sdist, then the resulting metadata can not be guaranteed until after it is indeed executed. We are talking about setuptools here. Other build backends (such as poetry) do not directly rely on an executable script, so things are much more static in the sdist.

the info is only useful if there's at least 1 dependency listed. Otherwise, it might be that there are no deps. (Which is good info), or simply that they're not specified, ie due not not having a wheel. A simple fix would be for the PyPi requires_dist field to return different things for these cases. Perhaps an empty list if no deps, null etc if not specified. That alone would let Poetry etc reduce the cases it has to download and build for.

Ah, I didn't know that. I never really looked at that JSON API. That might explain why pip does not use it (which I thought was odd).

Answer 68 · 2020-10-27T20:59:49.000Z

I'm sorry - I'd completely forgotten about the sys_platform flag. I do account for that in Pyflow. So, it is specified on PyPi for packages built with wheels, and package managers like Poetry can use this info.

Answer 69 · 2020-10-28T12:34:32.000Z

The fix is here: use wheels! Anyone (you) can go help any project to help them build wheels and upload them to PyPI. I do not think this is PyPI's role to intervene here. What other solution do you have in mind?

Usually in these situations what works best is lobbying for a better (simpler/clearer/broader/...) standards to be promoted by the authority. In this case I guess its PyPA? Or maybe both PSF and PyPA?

I never worked in any project that actively creates python packages so I might be wrong, but from my point of view packaging is not something that is currently crystal clear. It seems like there are way too many ways of doing one thing, so people like me don't really now what they should do. Because in the end, we have no idea about our choices impacts. Which, ultimately, leads to the problems you are talking here.

For what its worth, I find poetry's documentation to be a good way of preaching for better solutions. It's so clean it makes you want to make things cleaner.

From the user perspective its already a very good improvement to have a standards such as PEP 518 -- Specifying Minimum Build System Requirements for Python Projects, but that's apparently not sufficient? Or maybe the problem you are debating here only arise for older projects ?

What about this news: New pip resolver to roll out this year?

Answer 70 · 2020-10-28T13:42:30.000Z

@cglacet

Usually in these situations what works best is lobbying for a better (simpler/clearer/broader/...) standards to be promoted by the authority. In this case I guess its PyPA? Or maybe both PSF and PyPA?

Yes. That would be PyPA. They know all about these kinds of issues. They are actively working on solving them. These things take time. There is no need to lobby. There is need to participate with writing good documentation and good code. And most important of all, donate to fund developers to work full time on it.

What about this news: New pip resolver to roll out this year?

This is a part of the work, yes. Once this rolls out, PyPA will be able to move on to solving other packaging issues. This work was partly done thanks to financial grants (money donations).

You can read more about related, ongoing work (these links are only a short, semi-random selection, but they are all somewhat intertwined):

I never worked in any project that actively creates python packages so I might be wrong, but from my point of view packaging is not something that is currently crystal clear. It seems like there are way too many ways of doing one thing, so people like me don't really now what they should do. Because in the end, we have no idea about our choices impacts. Which, ultimately, leads to the problems you are talking here.

Yes. From my point of view, issue is that the overwhelming majority of advice found on the internet (articles, blogs, StackOverflow answers, etc.) is either outdated, misguided, or plain wrong.

A good reference is this website (from PyPA itself):

https://packaging.python.org/

If you follow poetry's workflows you are already in very good hands, and you should not worry about anything too much. Upload wheels! Well, you need to upload both sdists and wheels. The sdists are still very important, do not forget them.

For what its worth, I find poetry's documentation to be a good way of preaching for better solutions. It's so clean it makes you want to make things cleaner.

Yes, it is also doing a very good job at getting rid of outdated, bad practices.

[Sadly somehow, there are always users pushing for poetry to adapt to their own broken workflows, instead of users changing their habits for the clean workflows of poetry. It is a constant battle.]

From the user perspective its already a very good improvement to have a standards such as PEP 518 -- Specifying Minimum Build System Requirements for Python Projects, but that's apparently not sufficient? Or maybe the problem you are debating here only arise for older projects ?

Yes, this was another great step forward. Python packaging ecosystem is improving a lot these days.

And yes, exactly, a great hurdle is keeping the compatibility with older projects. This slows down the work a lot. In particular older, broken setuptools / distutils setup.py based projects are very problematic. Although it is nowadays entirely possible to write clean, well-behaved, and standard-conform setuptools based projects.

[I am writing this of the top of my head according to the bits of info I have gathered here and there along the way. I do not have insight into all the processes involved, so there might be some inaccuracies. Feel free to correct me. Feel free to ask me for clarifications.]

Answer 71 · 2020-10-28T15:10:05.000Z

If you are looking for projects to help contribute to that don't yet have wheels, this site lists the top 360 packages, a handful of which don't have wheels: https://pythonwheels.com/

Answer 72 · 2020-10-30T08:28:44.000Z

The discussion on python.org might be interesting for some as well: Standardized way for receiving dependencies

Answer 73 · 2020-11-04T08:09:12.000Z

Upload a poetry.lock to CI will avoid resolving dependencies.

Answer 74 · 2020-11-12T13:34:46.000Z

Here is a toml cost more than 700 secs in 'resolving dependencies'

[[tool.poetry.source]]
name = "aliyun"
url = "https://mirrors.aliyun.com/pypi/simple/"
default = true
[tool.poetry]
name = "omega"
version = "0.7.0"
description = "Blazing fast data server for Zillionare"
authors = ["jieyu <code@jieyu.ai>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
apscheduler = "^3.6"
arrow = "^0.15"
cfg4py = "^0.6"
"ruamel.yaml" = "^0.16"
aioredis = "^1.3"
hiredis = "^1.0"
numpy = "^1.18"
aiohttp = "^3.6"
pytz = "^2020.1"
xxhash = "^1.4"
zillionare-omicron = "0.2.0"
aiocache = "^0.11"
sanic = "^20.3"
psutil = "^5.7"
termcolor = "^1.1"
gino = "^1.0"
asyncpg = "^0.20"
sh = "^1.13"

[tool.poetry.dev-dependencies]
flake8 = "^3.8.4"
flake8-docstrings = "^1.5.0"
tox = "^3.14"
coverage = "^4.5.4"
Sphinx = "^1.8.5"
black = "^20.8b1"
pre-commit = "^2.8.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

No difference with/without private repo defined. Hope helps.

Answer 75 · 2020-11-12T13:40:19.000Z

One thing @zillionare's example has in common with mine is black. They recently had a bug so they pulled off the wheel from PyPI: https://pypi.org/project/black/20.8b1/#files. This could be contributing to this.

psf/black#1669

Answer 76 · 2020-11-12T15:05:50.000Z

One thing @zillionare's example has in common with mine is black. They recently had a bug so they pulled off the wheel from PyPI: https://pypi.org/project/black/20.8b1/#files. This could be contributing to this.

psf/black#1669

I did the test, sounds like to me blakc 20.8b1 is not the culprit.

My steps:

change to aliyun repo since I'm in china mainland
clear cache and remove peotry.lock
run poetry install, it takes 239.2s resolving dependencies.
clear cache and remove peorty.lock
comment out black = xxx from pyproject.toml
run poetry install, it takes 236.2 s resolving dependencies.

I'm new to poetry, so I'm not sure if my test steps are right. And sounds to me clear cache is not nessecary.

---- update 2020/11/22
Today I tried on another machine:

it's still running...

Answer 77 · 2020-12-02T13:49:02.000Z

@zillionare Have went through the same issue, fixed it after setting up VPN to kill GFW.

Answer 78 · 2020-12-09T14:43:23.000Z

@zillionare Have went through the same issue, fixed it after setting up VPN to kill GFW.

Guess this is the root cause for my case too. I have setup a proxy for pypi, it still runs slow (with -vvv options on, I can see it's progressing), and I know how it works now. A lot of files need to be download before the deps is resolved.

Hope poetry can support mirror pip source, so the performance issue will be solved.

@cglacet

Answer 79 · 2021-02-04T22:21:32.000Z

What if we create a service from pydeps which can take millions of requests from around the world, then make poetry to use that service to resolve dependencies?

Answer 80 · 2021-02-16T19:42:09.000Z

I had this issue out of nowhere on Mac OS X. As funny as it sounds, rebooting the computer solved the issue.

Answer 81 · 2021-02-17T19:18:39.000Z

I am writing this comment while poetry is resolving dependencies. 1338 seconds so far, and counting. Running poetry 1.1.4.

Ran poetry init, setting no dependencies during setup (python 3.8.7, linux/debian). Then ran poetry add meltano. I'm on a 30Mbps connection both ways (no firewall, I'm in Portugal). Not the fastest connection, but this is simply unusable. Interestingly, meltano has a pyproject.toml and poetry.lock so I'd expect the dependency resolution to be super fast.
1727 seconds...

Answer 82 · 2021-02-17T20:05:38.000Z

After 1800 seconds, I killed the above, and reran it as poetry add -vvv meltano. The output is at https://gist.github.com/laurentS/53ae55faa3d19d9934f46bc40864bb60 though I killed that one after about 1000 seconds.
botocore seems to be causing problems, and its dependencies are resolved 85+ times. Looking even quickly at the logs, poetry is downloading 60+ versions of botocore and dependencies. Why is this happening, particularly again, as meltano already has a poetry.lock file?

Just to confirm, my pyproject.toml contains no other dependencies, this is a clean virtual env made by poetry.

Answer 83 · 2021-02-18T20:12:40.000Z

I'm having issues with Poetry being extremely slow as well. With a completely blank project, running poetry add Django@~2.2.18 takes 12.5 minutes with the vast majority of that time spent resolving django's 2 dependencies.

I ran poetry install on our production Django project with ~70 top-level dependencies and left it running overnight but it still didn't finish. I understand these issues aren't necessarily the fault of Poetry but it doesn't seem like there's a way to make this tool usable for larger projects given the state of the python ecosystem.

Answer 84 · 2021-02-18T20:48:04.000Z

You can always review your direct dependencies and put stricter version ranges. This way, the dependency resolver has less combinations to try and can succeed (or fail) quicker.

Not usually a great idea to list indirect dependencies, but if it helps the dependency resolver, you might also consider adding those indirect dependencies with a strict version range in your pyprojct.toml as well.

If poetry add Django@~2.2.18 on a blank project takes minutes, then something else must be seriously wrong, that seems very suspicious.

[Aside: what project has 70 top-level dependencies? All of this looks like red flags to me. I can't help but think that maybe such big projects come from companies with the resources to pay developers and solve such issues themselves instead of pushing the ball towards small volunteer-run tools. I say this, because people just keep adding their complaint without providing any new insight. The problem is known, it has been discussed at length, many root causes have been explained multiple times in this thread. We are running in circles here.]

Answer 85 · 2021-02-18T21:12:26.000Z

If this isn't an issue that Poetry wants to fix and additional input or examples are not welcome, maybe it's best to close it. There are comments further up the thread asking for examples, so I assumed it would be fine to add mine here. If this is just how to tool behaves due to circumstances outside of Poetry's control then I think it might be worth adding a larger disclaimer in the documentation than the current FAQ question because it wasn't clear to me as a potential user of this tool that it might not work for all use cases.

consider adding those indirect dependencies

I can do that, but at that point there's not much of a benefit to using Poetry over a plain requirements.txt file.

If poetry add Django@~2.2.18 on a blank project takes minutes, then something else must be seriously wrong, that seems very suspicious.

That's what I was trying to get at. If this simple of a use case is so slow then clearly this tool won't work well for a non-trivial project. Maybe there's an issue with my environment or with Poetry itself and someone else can try and see if they can replicate what I'm seeing.

what project has 70 top-level dependencies

I don't think that's unusual, my project is a small django backend (5 devs, not a large company with endless resources) and 20+ of those dependencies are django plugins.

Answer 86 · 2021-02-18T21:20:59.000Z

If poetry add Django@~2.2.18 on a blank project takes minutes, then something else must be seriously wrong, that seems very suspicious.

That's what I was trying to get at. If this simple of a use case is so slow then clearly this tool won't work well for a non-trivial project. Maybe there's an issue with my environment or with Poetry itself and someone else can try and see if they can replicate what I'm seeing.

Provide meaningful details: python interpreter type, version, CPU type and bitness, OS type and version, poetry config, potential relevant network info (vpn, pypi mirrors), etc.
I just tried the exact same command on a blank project before writing my previous message and it finished in seconds. Have you tried maybe deleting the poetry cache entirely?

Answer 87 · 2021-02-18T21:53:01.000Z

Python 3.7.9 on intel-based macOS 11.2.1 installed via pyenv
No VPN or pypi mirrors
Ran poetry cache clear pypi --all prior to poetry add and the issue persisted

$ poetry config --list
cache-dir = "/Users/me/Library/Caches/pypoetry"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/me/Library/Caches/pypoetry/virtualenvs

Answer 88 · 2021-02-18T22:03:02.000Z

Seems like a pretty standard configuration (except the macos 11 big sur part, which provided lots of headaches recently, maybe it's solved by now, I lost track). Maybe the verbose output could provide some insight: poetry add -vvv Django@~2.2.18 into what poetry is struggling with.

As a point of comparison, how does poetry run python -m pip install 'Django~=2.2.18' perform?

Answer 89 · 2021-02-18T22:37:07.000Z

Thank you for helping me look into this!

The timing seems to vary dramatically between runs, I ran poetry add a few times and sometimes it hangs for several minutes after printing Using virtualenv: ... and sometimes it proceeds in a few seconds. These are logs from a run where it didn't stall with some timing info added using my phone stopwatch.

$ time poetry run python -m pip install 'Django~=2.2.18'
# Executed in 34.74 secs

$ poetry cache clear pypi --all
$ poetry add -vvv Django@~2.2.18
Using virtualenv: /Users/alexgrover/Developer/poetry-test/.venv         <------ 0:00
PyPI: No release information found for django-1.0.1, skipping
PyPI: No release information found for django-1.0.2, skipping
PyPI: No release information found for django-1.0.3, skipping
PyPI: No release information found for django-1.0.4, skipping
PyPI: No release information found for django-1.1, skipping
PyPI: No release information found for django-1.1.1, skipping
PyPI: No release information found for django-1.1.2, skipping
PyPI: 1 packages found for django >=2.2.18,<2.3.0

Updating dependencies
Resolving dependencies...
   1: fact: poetry-test is 0.1.0
   1: derived: poetry-test
   1: fact: poetry-test depends on Django (~2.2.18)
   1: selecting poetry-test (0.1.0)
   1: derived: Django (~2.2.18).                     <------ prints up to here at 0:17
PyPI: No release information found for django-1.0.1, skipping
PyPI: No release information found for django-1.0.2, skipping
PyPI: No release information found for django-1.0.3, skipping
PyPI: No release information found for django-1.0.4, skipping
PyPI: No release information found for django-1.1, skipping
PyPI: No release information found for django-1.1.1, skipping
PyPI: No release information found for django-1.1.2, skipping
PyPI: 1 packages found for django >=2.2.18,<2.3.0
PyPI: Getting info for django (2.2.18) from PyPI
   1: fact: django (2.2.18) depends on pytz (*)
   1: fact: django (2.2.18) depends on sqlparse (>=0.2.2)
   1: selecting django (2.2.18)
   1: derived: sqlparse (>=0.2.2)
   1: derived: pytz (*)                                     <------ 0:24
PyPI: 7 packages found for sqlparse >=0.2.2
PyPI: 86 packages found for pytz *
PyPI: Getting info for sqlparse (0.4.1) from PyPI             <------ 0:41
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: sqlparse-0.4.1-py3-none-any.whl         <------ 4:25
   1: selecting sqlparse (0.4.1)
PyPI: Getting info for pytz (2021.1) from PyPI           <------ 5:44
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: pytz-2021.1-py2.py3-none-any.whl              <------ 8:15
   1: selecting pytz (2021.1)
   1: Version solving took 556.392 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 3 installs, 0 updates, 0 removals.         <------ 9:38

  • Installing pytz (2021.1): Downloading... 100%
  • Installing pytz (2021.1): Installing...
  • Installing pytz (2021.1)
  • Installing sqlparse (0.4.1): Downloading... 100%
  • Installing sqlparse (0.4.1): Installing...
  • Installing sqlparse (0.4.1)
  • Installing django (2.2.18): Downloading... 100%
  • Installing django (2.2.18): Installing...
  • Installing django (2.2.18)

<------ 10:56

Answer 90 · 2021-02-18T23:30:17.000Z

Those numbers are very surprising. There does not seem to be anything particularly difficult happening here. It does not seem to be the dependency resolver that is struggling (doing lots of back-tracking or things like that). I wonder what it could be... Maybe open a separate bug ticket, I feel like this issue might be different than the rest of the thread.

Answer 91 · 2021-02-18T23:36:51.000Z

Here's another fun one: 1: Version solving took 3340.497 seconds.

I didn't think there was much controversial about the requirements:

python = "^3.9"
bokeh = "^=2.2.3"
colorcet = "^2.0.2"
matplotlib = "^3.3.3"
numpy = "^1.19.5"
pandas = "^1.2.0"
shapely = "^1.7.1"
pycryptodome = "^3.9.9"
PyYAML = "^5.3.1"
scipy = "^1.6.0"
tqdm = "^4.55.2"

I need to use custom CA certificate due to the companies VPN, but I don't know if that is causing the slowness.

poetry.lock.txt
pyproject.toml.txt

Answer 92 · 2021-02-18T23:47:54.000Z

You can always review your direct dependencies and put stricter version ranges. This way, the dependency resolver has less combinations to try and can succeed (or fail) quicker.

Not usually a great idea to list indirect dependencies, but if it helps the dependency resolver, you might also consider adding those indirect dependencies with a strict version range in your pyprojct.toml as well.

@sinoroc what do you suggest as a way forward, as a user of poetry? I understand that poetry is trying to do things better in the packaging world (I use it for my own library, it's brilliant), but I also cannot rely on a tool that takes 30+ minutes to (not) resolve 1 single package (see my example above, meltano with logs linked, which pip installed in under 2 minutes with all dependencies). If I need to do dependency resolution by hand, there is no point using a tool.

I'm a solo developer, also maintaining a couple of opensource projects, I understand there are only so many hours in the day, and it definitely isn't my goal to demand that you guys fix this, just trying to understand what can be done. And obviously happy to help with providing more details on failing examples.

If you have a fairly standard set of tricks to bypass the problem, maybe it would be worth putting them in the docs, or as a pinned issue?

Answer 93 · 2021-02-19T14:04:20.000Z

I am facing similar problem when adding a single package compliance-trestle to a newly initialised project.

Please note there are no other packages specified in this project.

command poetry add compliance-trestle was running forever, therefore I have created a log file (generated using poetry add compliance-trestle -vv > add.log 2&> 1 command ) which I am attaching for your reference.

add.log

Answer 94 · 2021-02-19T14:58:59.000Z

I am facing similar problem when adding a single package compliance-trestle to a newly initialised project.

Please note there are no other packages specified in this project.

command poetry add compliance-trestle was running forever, therefore I have created a log file (generated using poetry add compliance-trestle -vv > add.log 2&> 1 command ) which I am attaching for your reference.

add.log

In this case I also wanted try and see if pipenv could resolve help me.. and yes it did indeed help...

by bailing out quickly
telling me the cause of failure.. (though not the same as what poetry would encounter) [ see below ]

Installing compliance-trestle...
Adding compliance-trestle to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/utils.py", line 1395, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/Users/pritam/.pyenv/versions/3.8.0/lib/python3.8/site-packages/pipenv/utils.py", line 833, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches black>=19.10b0 (from datamodel-code-generator[http]==0.8.1->compliance-trestle==0.7.2->-r /var/folders/vq/f2j7762d1q5cx1wfgfk_5rkm0000gn/T/pipenv31uibkngrequirements/pipenv-o1s54rc7-constraints.txt (line 2))
Skipped pre-versions: 18.3a0, 18.3a0, 18.3a1, 18.3a1, 18.3a2, 18.3a2, 18.3a3, 18.3a3, 18.3a4, 18.3a4, 18.4a0, 18.4a0, 18.4a1, 18.4a1, 18.4a2, 18.4a2, 18.4a3, 18.4a3, 18.4a4, 18.4a4, 18.5b0, 18.5b0, 18.5b1, 18.5b1, 18.6b0, 18.6b0, 18.6b1, 18.6b1, 18.6b2, 18.6b2, 18.6b3, 18.6b3, 18.6b4, 18.6b4, 18.9b0, 18.9b0, 19.3b0, 19.3b0, 19.10b0, 19.10b0, 20.8b0, 20.8b1
There are incompatible versions in the resolved dependencies:

I figured pipenv fails installing black>=19.10b0, whereas poetry succeeds in it (as i learnt)..
As next step I tried installing datamodel-code-generator -E http and it worked flawlessly using poetry
and with datamodel-code-generator installed poetry add compliance-trestle completed in 2 seconds...

Based on this experience what I seek from poetry is to
a) of course being able to resolve.. and
b) if it is failing .. fail fast with some directions to look into..

Thank you for a wonderful product btw 👍

Answer 95 · 2021-02-19T17:41:54.000Z

[I am not a maintainer]

My recommendation to pin or restrict the version ranges of some dependencies, or even some indirect dependencies, is just a workaround to help the dependency resolution algorithm in cases where it is struggling to find a suitable combination of distributions to install. If/when poetry's dependency resolution gets better those version pins and restrictions could probably be removed.

Otherwise, there are some things that maybe help (or maybe not, hard to tell, since there are so many cases presented here, and I'm not even sure they are all due to dependency resolution):

check that the python restriction in your pyproject.toml is compatible with your dependencies (I think I remember seeing quite some cases where it would lead to unsolvable dependencies, I could try to find those again to show you what I'm talking about)
clear poetry's cache
build your own index server containing only the dependencies you need (so that the dependency resolution only considers a very limited amount of distributions)
check your network traffic (make sure that it is not a download speed issue)
try different Python versions (maybe some versions are easier to solve)
help reviewing/improving the code of the dependency resolution algorithm
get in touch with the maintainers and see how to support them (financially, etc.)

Answer 96 · 2021-02-20T19:59:15.000Z

I wonder if maybe some of you are finding themselves in a situation similar to this:

The current project's Python requirement (>=3.6) is not compatible with some of the required packages Python requirement:
    - isort requires Python >=3.6,<4.0, so it will not be satisfied for Python >=4.0

  Because isort (5.7.0) requires Python >=3.6,<4.0
   and no versions of isort match >5.7.0,<6.0.0, isort is forbidden.

For project with few dependencies this fails quickly, but I could also imagine that with lots of dependencies it could be resolving for what seems forever. Might be totally unrelated, might be a false lead. But maybe it's worth looking into this...

In your pyproject.toml set the Python requirement to a fix major-minor and see if it helps the dependency resolution. For example:

[tool.poetry.dependencies]
python = "3.6"

instead of >=3.6 or ^3.6, etc.

Answer 97 · 2021-02-26T20:49:10.000Z

This is perhaps a naive question, but why can't we just not do dependency resolution? Since pip doesn't seem to do this and works just fine, why does Poetry do it?

Answer 98 · 2021-02-26T21:26:34.000Z

Here’s a pretty good summary of why pip and requirements.txt isn’t sufficient for many use cases: https://modelpredict.com/wht-requirements-txt-is-not-enough

Answer 99 · 2021-02-26T22:06:40.000Z

This is perhaps a naive question, but why can't we just not do dependency resolution? Since pip doesn't seem to do this and works just fine, why does Poetry do it?

Correction: pip used to "not do dependency resolution". And it was often not working fine, that was probably one of the main reasons why people wrote poetry and migrated to it.

pip now has an actual dependency resolution algorithm since some months, I can't find the actual release date and version number. Looks like it's 20.3 (2020-11-30):

Answer 100 · 2021-03-10T15:07:05.000Z

For me a fresh installation of poetry in python 3.8 (setup via pyenv) takes forever to do poetry add trio in a new project. Logs:

PyPI: 21 packages found for trio *
Using version ^0.18.0 for trio

Updating dependencies
Resolving dependencies...
   1: fact: trio-statemachines is 0.1.0
   1: derived: trio-statemachines
   1: fact: trio-statemachines depends on trio (^0.18.0)
   1: selecting trio-statemachines (0.1.0)
   1: derived: trio (>=0.18.0,<0.19.0)
PyPI: 1 packages found for trio >=0.18.0,<0.19.0
   1: fact: trio (0.18.0) depends on attrs (>=19.2.0)
   1: fact: trio (0.18.0) depends on sortedcontainers (*)
   1: fact: trio (0.18.0) depends on async-generator (>=1.9)
   1: fact: trio (0.18.0) depends on idna (*)
   1: fact: trio (0.18.0) depends on outcome (*)
   1: fact: trio (0.18.0) depends on sniffio (*)
   1: fact: trio (0.18.0) depends on cffi (>=1.14)
   1: selecting trio (0.18.0)
   1: derived: cffi (>=1.14)
   1: derived: sniffio
   1: derived: outcome
   1: derived: idna
   1: derived: async-generator (>=1.9)
   1: derived: sortedcontainers
   1: derived: attrs (>=19.2.0)
PyPI: 6 packages found for cffi >=1.14
PyPI: 4 packages found for sniffio *
PyPI: 4 packages found for outcome *