astral-sh/rye

Better uv integration

mitsuhiko opened this issue · 22 comments

mitsuhiko commented

This is a meta task to look into actually leveraging uv properly.

  • lock with the lower bound of requires-python rather than the current venv version
  • make all --prerelease options available
  • make all --resolution modes available
  • #676
brunorpinho commented

Is there a config option to start using rye with uv instead of pip today?

golgor commented

@brunorpinho
See the documentation about the configuration:

# When set to `true` enables experimental support of uv as a replacement
# for pip-tools. Learn more about uv here: https://github.com/astral-sh/uv
use-uv = false

This was also asked during the installation and I have been using it and testing it out today. No real heavy testing, but it works great for me so far.

  • ❤️2
mitsuhiko commented

Note that for this to work on windows you need the not yet released version as it's disabled for safety reasons on windows still since I did not have time yet to verify basic functionality.

mjclarke94 commented

One that might be worth including here for feature completeness is the dependency overrides functionality. A common complaint with poetry is when you have a chain of git repositories dependant on one another (say, pointing at main), an update to one repo breaks everything downstream of it as poetry locks to a specific commit hash.

The overrides functionality in uv could be a fairly lightweight way to bypass this whilst all the repos are in active development.

  • 👍6
sbarrios93 commented

Does using uv not take into account sources listed in pyproject.toml?

I have the following on my pyproject.toml file:

[[tool.rye.sources]]
name = "pytorch"
type = "index"
url = "https://download.pytorch.org/whl/nightly/cu121"

The lastest stable release of torch is 2.2.0 but I'm trying to download 2.3.0 which is prerelease.

Does not work:

❯ rye add torch
Added torch>=2.2.0 as regular dependency

Does not work:

❯ rye add --pre torch
Added torch>=2.2.0 as regular dependency

Seems like rye is constructing a similar command argument as the following. Based on the add.rs file

❯ VIRTUAL_ENV=.venv uv pip compile --no-deps --no-header --python-version 3.11.7 --prerelease=allow -
torch
Resolved 1 package in 1.54s
torch==2.2.0

But if try to install directly with uv, it does work:

❯ uv pip install torch --prerelease allow --index-url https://download.pytorch.org/whl/nightly/cu121
Resolved 22 packages in 570ms
Downloaded 11 packages in 28.06s
Installed 22 packages in 117ms
 + filelock==3.9.0
 + fsspec==2023.4.0
 + jinja2==3.1.2
 + markupsafe==2.1.3
 + mpmath==1.2.1
 + networkx==3.0rc1
 + nvidia-cublas-cu12==12.1.3.1
 + nvidia-cuda-cupti-cu12==12.1.105
 + nvidia-cuda-nvrtc-cu12==12.1.105
 + nvidia-cuda-runtime-cu12==12.1.105
 + nvidia-cudnn-cu12==8.9.2.26
 + nvidia-cufft-cu12==11.0.2.54
 + nvidia-curand-cu12==10.3.2.106
 + nvidia-cusolver-cu12==11.4.5.107
 + nvidia-cusparse-cu12==12.1.0.106
 + nvidia-nccl-cu12==2.19.3
 + nvidia-nvjitlink-cu12==12.1.105
 + nvidia-nvtx-cu12==12.1.105
 + pytorch-triton==3.0.0+901819d2b6
 + sympy==1.11.1
 + torch==2.3.0.dev20240219+cu121
 + typing-extensions==4.8.0
  • 👍4
mitsuhiko commented

@sbarrios93 yes, looks like a bug. I will fix that.

  • 👍2
sbarrios93 commented

@mitsuhiko, I'm assuming that a fix would involve setting the default source in config.toml for rye (usually PyPi) as --index-url, with any additional index sources as --extra-index-url. However, based on astral-sh/uv#1451 (comment), I'm guessing that such a fix on rye's side wouldn't fix the resolution issue.

I've added some insights specific to PyTorch in astral-sh/uv#1723, which I closed in favor of astral-sh/uv#1451.

I don't want to hijack this thread – let me know if it's better to open a separate issue for this discussion.

osuchw commented

rye add with --features currently does not work when uv is enabled.

$ rye --version
rye 0.25.0
commit: 0.25.0 (d8e00cea1 2024-02-19)
platform: linux (x86_64)
self-python: cpython@3.12
symlink support: true
uv enabled: true

$ rye add flask --features dotenv
error: failed to resolve packages:
thread 'main' panicked at crates/uv-resolver/src/resolution.rs:136:29:
Every package should have metadata: NameVersion(PackageName("flask"), "3.0.2")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

$ rye config --set-bool behavior.use-uv=false

$ rye add flask --features dotenv
Added flask[dotenv]>=3.0.2 as regular dependency

Installing with uv by itself works.

$ uv --version
uv 0.1.6
$ uv venv
Using Python 3.12.1 interpreter at /home/osuchw/.rye/py/cpython@3.12.1/install/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate

$ . .venv/bin/activate

$ uv pip install flask[dotenv]
Resolved 8 packages in 316ms
Downloaded 8 packages in 83ms
Installed 8 packages in 17ms
 + blinker==1.7.0
 + click==8.1.7
 + flask==3.0.2
 + itsdangerous==2.1.2
 + jinja2==3.1.3
 + markupsafe==2.1.5
 + python-dotenv==1.0.1
 + werkzeug==3.0.1
  • 👍1
mitsuhiko commented

This at the very least is triggering a weird code path in uv. I filed it there for now: astral-sh/uv#1801

Could be trivially worked around on the rye side though if needed.

osuchw commented

Thank you for looking into it. There is no more panic with rye 0.26.0 but --features is being ignored.
There is #745 now with more details.

utkarshgupta137 commented

Not sure if this is the right place, but I can't figure out an easy way to get rye (with uv enabled) to update my rust deps.
Currently (rye 0.29.0, no uv), when I run rye sync --no-lock, it re-installs all workspace members, including re-building all the rust dependencies:

$ rye sync --no-lock -v
Python version already downloaded. Skipping.
Reusing already existing virtualenv
Installing dependencies
Using pip-tools configuration defaults found in 'pyproject.toml'.
Found existing installation: proj 0.1.0
Uninstalling proj-0.1.0:
  Successfully uninstalled proj-0.1.0
Found existing installation: proj-entrypoint 0.1.0
Uninstalling proj-entrypoint-0.1.0:
  Successfully uninstalled proj-entrypoint-0.1.0
Found existing installation: proj-toolbox 0.1.0
Uninstalling proj-toolbox-0.1.0:
  Successfully uninstalled proj-toolbox-0.1.0
Looking in indexes: https://pypi.org/simple/
Obtaining file:///lib/proj-entrypoint (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 1))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///lib/proj-toolbox (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 2))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining file:///src (from -r /var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/tmp4wq0kjth (line 3))
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: proj-entrypoint, proj-toolbox, proj
  Building editable for proj-entrypoint (pyproject.toml) ... done
  Created wheel for proj-entrypoint: filename=proj_entrypoint-0.1.0-cp312-cp312-macosx_11_0_arm64.whl size=1004 sha256=fdf8bc908d40ca6bb2bf4b302790cd34f4fef726aa9e5321e325b03807ac5e2b
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/10/c1/9a/4b146280476e5340ceac7f2cc7fe6722d555ff0209eabc2f15
  Building editable for proj-toolbox (pyproject.toml) ... done
  Created wheel for proj-toolbox: filename=proj_toolbox-0.1.0-cp312-cp312-macosx_11_0_arm64.whl size=999 sha256=c896f9950182d5fec3d9171a3bf2b9bf20364b40345b046ed2dd15506d168bc2
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/d2/12/22/425ebf9a3f9b82245195a3d48f64aaa3c63d365f24108c023d
  Building editable for proj (pyproject.toml) ... done
  Created wheel for proj: filename=proj-0.1.0-0.editable-py3-none-any.whl size=2649 sha256=549df1a8990cbb11dd8d2d09e1c8a9e14f4e9c02405745d08c2ca8209e0972b1
  Stored in directory: /private/var/folders/9n/vtv29bd54r10hvbr7z0tdr780000gn/T/pip-ephem-wheel-cache-kolmdpsj/wheels/be/16/c1/367c544384017c548a18cd5b8ec9d158b60362a7ae1a22dff3
Successfully built proj-entrypoint proj-toolbox proj
Installing collected packages: proj-toolbox, proj-entrypoint, proj
Successfully installed proj-0.1.0 proj-entrypoint-0.1.0 proj-toolbox-0.1.0
Done!

With uv enabled, I get the following:

$ rye sync --no-lock -v
Python version already downloaded. Skipping.
Reusing already existing virtualenv
Installing dependencies
    0.002595s DEBUG uv_client::registry_client Using registry request timeout of 300s
 uv::requirements::from_source source=/Users/utkarsh/work/rye-proj/requirements-dev.lock
    0.137214s DEBUG uv_interpreter::python_environment Found a virtualenv through VIRTUAL_ENV at: /Users/utkarsh/work/rye-proj/.venv
    0.138290s DEBUG uv_interpreter::interpreter Cached interpreter info for Python 3.12.1, skipping probing: /Users/utkarsh/work/rye-proj/.venv/bin/python
    0.138299s DEBUG uv::commands::pip_sync Using Python 3.12.1 environment at /Users/utkarsh/work/rye-proj/.venv/bin/python
 uv_client::flat_index::from_entries
    0.144290s DEBUG uv_installer::plan Treating editable requirement as immutable: proj-entrypoint==0.1.0 (from file:///Users/utkarsh/work/rye-proj/lib/proj-entrypoint)
    0.144330s DEBUG uv_installer::plan Treating editable requirement as immutable: proj-toolbox==0.1.0 (from file:///Users/utkarsh/work/rye-proj/lib/proj-toolbox)
    0.144332s DEBUG uv_installer::plan Treating editable requirement as immutable: proj==0.1.0 (from file:///Users/utkarsh/work/rye-proj/src)
    0.144366s DEBUG uv_installer::plan Requirement already satisfied: alabaster==0.7.16
    0.144369s DEBUG uv_installer::plan Requirement already satisfied: asciitree==0.3.3
    0.144370s DEBUG uv_installer::plan Requirement already satisfied: asttokens==2.4.1

Where proj-entrypoint & proj-toolbox are rust crates & proj depends on both of them.
It would be nice if I could pass a flag to force rye to re-build my rust dependencies or better yet, have it automatically figure it out.

geoqiao commented

Will rye add -r requirements.txt be supported, just like uv pip install -r requirements.txt or something like that?

It's needed when clone some projects which not init by rye

bluss commented

@geoqiao Rye init already supports importing requirements files

Edit: Indeed, as you say, it seems like it's not documented.

  • 👍1
bluss commented

I'll suggest here that uv's --offline flag could be made available on rye lock and rye sync.

geoqiao commented

@geoqiao Rye init already supports importing requirements files

Sorry, I didn't find in doc. Could you teach me how to do that? Thanks!

geoqiao commented

@geoqiao Rye init already supports importing requirements files

Thanks for your reply, I found the -r option in the command line section of the document

MilesCranmer commented

Friendly ping. Is there any way to pass --prerelease=allow to rye? I got hit with this today and literally can't install a package with rye because of it:

Reusing already existing virtualenv
Generating production lockfile: /Users/mcranmer/PermaDocuments/spock/requirements.lock
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of exoplanet-core==0.3.0rc2 and celmech==1.5.0 depends on exoplanet-core==0.3.0rc2, we can conclude that celmech==1.5.0 cannot be used.
      And because you require celmech==1.5.0, we can conclude that the requirements are unsatisfiable.

      hint: exoplanet-core was requested with a pre-release marker (e.g., exoplanet-core==0.3.0rc2), but pre-releases weren't enabled (try: `--prerelease=allow`)
error: could not write production lockfile for project

Caused by:
    Failed to run uv compile /var/folders/1h/xyppkvx52cl6w3_h8bw_gdqh0000gr/T/.tmpgCC8Dh/requirements.txt. uv exited with status: exit status: 1

but alas, I cannot seem to pass this option through to uv.

charliermarsh commented

@MilesCranmer - I can look into adding it. Note that you can work around this by telling uv to allow pre-releases for exoplanet-core specifically, by adding a requirement like:

exoplanet-core >= 0.3.0rc2

As long as you include some pre-release segment in the specifier, we'll enable pre-releases for that dependency.

  • 👍1
MilesCranmer commented

Awesome. Thanks!

bluss commented

@MilesCranmer have you had any success using rye sync --pre ?

  • 👍1
MilesCranmer commented

Yep, that did it, thanks! (Unfortunately it was a dependency of a dependency, so putting it as an explicit dependency wasn't optimal)

Maybe the error message could point out the --pre flag? Right now the error is from uv and only mentions the --prerelease one.

bzm3r commented

@MilesCranmer have you had any success using rye sync --pre ?

Thanks for the hint.

I found that the --pre flag can also be passed to rye add (not just rye sync). So that's nice.