`conda_python()` point to unexpected Python directory

Question

`conda_python()` point to unexpected Python directory

qbao96xb opened this issue a year ago · 9 comments

Hi, I recently tried to use reticulate to leverage my Python script in the R environment. Unfortunately, I encountered an annoying error when trying to set up Python utilizing the conda environment - the RStudio is continuously aborted and restarts whenever I try py_config(). The following context describes my situation:

Conda version

active environment : r_py_310
active env location : /user_path/opt/anaconda3/envs/r_py_310
conda version : 23.3.1
conda-build version : not installed
python version : 3.10.12.final.0
virtual packages : __archspec=1=arm64
                               __osx=13.5.2=0
                               __unix=0=0
base environment : /user_path/miniforge3  (writable)
conda av data dir : /user_path/miniforge3/etc/conda

Python version

pythonp path: /user_path/opt/anaconda3/envs/r_py_310/bin/python
python versions: Python 3.10.13

As you might notice, I tried to use a conda environment named r_py_310 installed by miniforge3 and utilizing Python version 3.10. This environment is initialized under the configuration of CONDA_SUBDIR=osx-arm64 conda create -n $@, which means it is subjected to be compatible with the Mac M1 chip (arm64 architect). I follow this tutorial to create the r_py_310 environment.

R session info

R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.5.2
other attached packages: reticulate_1.31

And here is how I detect the error, it seems to be partially similar to this issue 385.

Set Sys.setenv("RETICULATE_PYTHON"="user_path/opt/anaconda3/envs/r_py_310/bin/python")
Run library(reticulate)
Initialize the conda envrioment use_condaenv('r_py_310', conda = "/user_path/miniforge3/condabin/conda")
Check py_available(), which returns FALSE
Debug py_available(), it only points to the if (is_python_initialized()) return(.globals$py_config$available). To be honest I don't really understand what is the root cause.
I check py_discover_config(), which returns:

python:         /user_path/opt/anaconda3/envs/r_py_310/bin/python
libpython:     /user_path/opt/anaconda3/envs/r_py_310/lib/libpython3.10.dylib
pythonhome:     /user_path/opt/anaconda3/envs/r_py_310:/user_path/opt/anaconda3/envs/r_py_310
version:        3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ]
numpy:          /user_path/opt/anaconda3/envs/r_py_310/lib/python3.10/site-packages/numpy
numpy_version:  1.22.3

NOTE: Python version was forced by RETICULATE_PYTHON

However, when I check conda_python(), it returns "/user_path/Library/r-miniconda-arm64/envs/r-reticulate/bin/python", which is definitely the default Python path of r-reticulate virtual environment.
As long as I run any code related to Python, the RStudio crashes and restarts.

In conclusion, I think that it cannot initialize the conda environment and falls back to the default installed virtual environment of the reticulate package causing the conflict which leads to the crash of RStudio. I could not find any further information as the RStudio IDE crashed immediately without any explanation.

Hope you guys can help on this, to me it is really frustrating.

Answer 1 · 2023-09-14T13:10:54.000Z

Is the path you're supplying to RETICULATE_PYTHON and conda (the one starting with "user_path/opt/...) missing a starting slash? Or is it meant to be a path relative to the current working directory?

Answer 2 · 2023-09-14T13:13:57.000Z

Is the path you're supplying to RETICULATE_PYTHON and conda (the one starting with "user_path/opt/...) missing a starting slash? Or is it meant to be a path relative to the current working directory?

oh sorry my bad, it does have the starting slash in my issue, just a typo here on the issue description

Answer 3 · 2023-09-14T16:12:56.000Z

In general, py_discover_config() should return the same thing as py_config()... the only difference between them might be that py_config() could prompt you to create the r-reticulate venv when/if it reaches that point in the Order of Discovery

Answer 4 · 2023-09-15T01:41:21.000Z

the bad thing is RStudio starts crashing whenever I run any function even py_config() or py_eval

Answer 5 · 2023-09-15T11:32:57.000Z

Can you run in R terminal?
If you're seeing a segfault, can you please run R under lldb and provide a backtrace?

(( segfaults are sometimes unavoidable with conda, if possible I would encourage switching to a virtual environment )).

Answer 6 · 2023-09-18T02:25:32.000Z

Can you run in R terminal? If you're seeing a segfault, can you please run R under lldb and provide a backtrace?

(( segfaults are sometimes unavoidable with conda, if possible I would encourage switching to a virtual environment )).

Hi Tomasz,

Thank you for your waiting. The weird thing is that my code can run absolutely error-free in R Terminal. Well, I am starting to get confused right now. The executable code is provided as follows:

[R.app GUI 1.79 (8238) aarch64-apple-darwin20]

[Workspace restored from /user_path/.RData]
[History restored from /user_path/.Rapp.history]

> library(reticulate)
> use_condaenv('r_py_310', conda = "/user_path/miniforge3/condabin/conda")
> py_config()
python:         /user_path/opt/anaconda3/envs/r_py_310/bin/python
libpython:      /user_path/opt/anaconda3/envs/r_py_310/lib/libpython3.10.dylib
pythonhome:     /user_path/opt/anaconda3/envs/r_py_310:/Users/baotran/opt/anaconda3/envs/r_py_310
version:        3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ]
numpy:          /user_path/opt/anaconda3/envs/r_py_310/lib/python3.10/site-packages/numpy
numpy_version:  1.22.3

NOTE: Python version was forced by use_python() function
> py_eval("1+1")
[1] 2
> polars <- import("polars")
> main <- import_main()
> raw_dt <- polars$read_csv("/user_path/file_path/raw/olist_order_items_sellers_products_geos.csv")
> head(raw_dt)

Which returns:

shape: (6, 26)
┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐
│ order_id  ┆ order_ite ┆ product_i ┆ seller_id ┆ … ┆ order_del ┆ order_del ┆ customer_ ┆ order_st │
│ ---       ┆ m_id      ┆ d         ┆ ---       ┆   ┆ ivered_ca ┆ ivered_cu ┆ id        ┆ atus     │
│ str       ┆ ---       ┆ ---       ┆ str       ┆   ┆ rrier_dat ┆ stomer_da ┆ ---       ┆ ---      │
│           ┆ i64       ┆ str       ┆           ┆   ┆ e         ┆ te        ┆ str       ┆ str      │
│           ┆           ┆           ┆           ┆   ┆ ---       ┆ ---       ┆           ┆          │
│           ┆           ┆           ┆           ┆   ┆ str       ┆ str       ┆           ┆          │
╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡
│ 00018f77f ┆ 1         ┆ e5f2d52b8 ┆ dd7ddc04e ┆ … ┆ 2020-11-0 ┆ 2020-11-1 ┆ f6dd3ec06 ┆ delivere │
│ 2f0320c55 ┆           ┆ 02189ee65 ┆ 1b6c2c614 ┆   ┆ 9         ┆ 7         ┆ 1db4e3987 ┆ d        │
│ 7190d7a14 ┆           ┆ 8865ca93d ┆ 352b383ef ┆   ┆ 14:35:00  ┆ 16:04:24  ┆ 629fe6b26 ┆          │
│ 4bdd3     ┆           ┆ 83a8f     ┆ e2d36     ┆   ┆           ┆           ┆ e5cce     ┆          │
│ 000229ec3 ┆ 1         ┆ c777355d1 ┆ 5b51032ed ┆ … ┆ 2021-07-2 ┆ 2021-07-3 ┆ 6489ae5e4 ┆ delivere │
│ 98224ef6c ┆           ┆ 8b72b67ab ┆ dd242adc8 ┆   ┆ 4         ┆ 0         ┆ 333f3693d ┆ d        │
│ a0657da4f ┆           ┆ beef9df44 ┆ 4c38acab8 ┆   ┆ 12:36:48  ┆ 13:19:16  ┆ f5ad4372d ┆          │
│ c703e     ┆           ┆ fd0fd     ┆ 8f23d     ┆   ┆           ┆           ┆ ab6d3     ┆          │
│ 00024acbc ┆ 1         ┆ 7634da152 ┆ 9d7a1d34a ┆ … ┆ 2022-02-1 ┆ 2022-02-1 ┆ d4eb9395c ┆ delivere │
│ df0a6daa1 ┆           ┆ a4610f159 ┆ 505240900 ┆   ┆ 5         ┆ 9         ┆ 8c0431ee9 ┆ d        │
│ e931b0381 ┆           ┆ 5efa32f14 ┆ 6425275ba ┆   ┆ 13:28:00  ┆ 13:32:39  ┆ 2fce09860 ┆          │
│ 14c75     ┆           ┆ 722fc     ┆ 1c2b4     ┆   ┆           ┆           ┆ c5a06     ┆          │
│ 00042b26c ┆ 1         ┆ ac6c36230 ┆ df560393f ┆ … ┆ 2020-08-2 ┆ 2020-09-0 ┆ 58dbd0b2d ┆ delivere │
│ f59d7ce69 ┆           ┆ 68f30de03 ┆ 3a51e7455 ┆   ┆ 4         ┆ 6         ┆ 70206bf40 ┆ d        │
│ dfabb4e55 ┆           ┆ 045865e4e ┆ 3ab94004b ┆   ┆ 09:46:09  ┆ 16:42:31  ┆ e62cd34e8 ┆          │
│ b4fd9     ┆           ┆ 10089     ┆ a5c87     ┆   ┆           ┆           ┆ 4d795     ┆          │
│ 00048cc3a ┆ 1         ┆ ef92defde ┆ 6426d21ac ┆ … ┆ 2020-11-2 ┆ 2020-11-2 ┆ 816cbea96 ┆ delivere │
│ e777c65db ┆           ┆ 845ab8450 ┆ a402a131f ┆   ┆ 2         ┆ 7         ┆ 9fe5b689b ┆ d        │
│ b7d2a0634 ┆           ┆ f9d70c526 ┆ c0a5d0960 ┆   ┆ 11:05:55  ┆ 13:44:35  ┆ 39cfc97a5 ┆          │
│ bc1ea     ┆           ┆ ef70f     ┆ a3c90     ┆   ┆           ┆           ┆ 06742     ┆          │
│ 00054e843 ┆ 1         ┆ 8d4f2bb7e ┆ 7040e82f8 ┆ … ┆ 2021-06-1 ┆ 2021-06-2 ┆ 32e2e6ab0 ┆ delivere │
│ 1b9d76758 ┆           ┆ 93e6710a2 ┆ 99a04d1b4 ┆   ┆ 9         ┆ 5         ┆ 9e778d99b ┆ d        │
│ 08bcb819f ┆           ┆ 8f34fa83e ┆ 34b795a43 ┆   ┆ 01:07:48  ┆ 22:03:38  ┆ f2e0ecd48 ┆          │
│ b4a32     ┆           ┆ e7d28     ┆ b4617     ┆   ┆           ┆           ┆ 98718     ┆          │
└───────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘

Answer 7 · 2023-09-18T02:43:34.000Z

The problem was solved when I switched into virtualenv as per your recommendation, the context is as follows:

python:         /user_path/directory_path/venv_py310/bin/python3
libpython:      /user_path/opt/anaconda3/envs/r_py_310/lib/libpython3.10.dylib
pythonhome:     /user_path/directory_path/venv_py310:/user_path/directory_path/venv_py310
virtualenv:     /user_path/directory_path/venv_py310/bin/activate_this.py
version:        3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ]
numpy:           [NOT FOUND]

NOTE: Python version was forced by RETICULATE_PYTHON

There is only 1 inconvenient problem left, as you may see, the libpython, which sources from a previously created conda environment - r_py_310, does not share the installed packages & libraries with the virtualenv venv_py310. Thus I have to use py_install() to install all the needed package all over again, which obviously takes up more space & causes extra time for set-up.

Could you please introduce me to how to configure the shared option as there are multiple libraries that have been installed?

Answer 8 · 2023-09-18T12:44:43.000Z

It looks like you created the venv using a conda python. Can you please install python using

reticulate::install_python(), or
official installers from www.python.org/downloads, or
if you're on linux, https://github.com/rstudio/python-builds/

Then virtualenv_create() (and virtualenv_starter()) should automatically find the python installed on your system.

library(reticulate)

install_python("3.10:latest")
virtualenv_create("my-venv", version = "3.10")

Answer 9 · 2023-09-29T04:29:09.000Z

Hi Tomasz,

Just so you know – the problem is solved using your recommendation. I think the issue came from RStudio since it cannot address the correct Python path and pathlib, specifically in use_condaenv('r_py_310', conda = "/user_path/miniforge3/condabin/conda") command. As a result, the reticulate package has to fall back the default setting and use virtual env created by itself - r-reticulate thus causing conflict in the software.

Thank for your support.