Installation Issue

Question

Installation Issue

scottlyden opened this issue 2 years ago · 9 comments

After upgrading to TensorFlow v2.11.0 I can no longer use GPU. I've tried also installing from the development version.

I tried upgrading the version of CUDA and cuDNN to the versions I understand to be compatible with this TF version (cuda_11.2.0_460.89_win10.exe and cudnn-11.2-windows-x64-v8.1.0.77.zip, respectively).

Tellingly (maybe), when I 'hide' (by renaming) the file C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin\cudnn64_8.dll I get no error messages. It's as if TF is not even trying to hit the GPU at all.

I'm sorry to bother you with this, but I'm at my wits' end.

tf$config$list_physical_devices()
[[1]]
PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')

Session info:

> reticulate::py_config()
python:         C:/Miniconda/python.exe
libpython:      C:/Miniconda/python39.dll
pythonhome:     C:/Miniconda
version:        3.9.1 (default, Dec 11 2020, 09:29:25) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Miniconda/Lib/site-packages/numpy
numpy_version:  1.24.1
tensorflow:     C:\MINICO~1\lib\site-packages\tensorflow\__init__.p

python versions found: 
 C:/Miniconda/python.exe
 C:/Miniconda/envs/py310/python.exe
 C:/Miniconda/envs/test_python/python.exe
 C:/anaconda3/python.exe
> tensorflow::tf_config()
TensorFlow v2.11.0 (C:\MINICO~1\lib\site-packages\tensorflow\__init__.p)
Python v3.9 (C:/Miniconda/python.exe)
> reticulate::import("tensorflow")
Module(tensorflow)
> reticulate::py_last_error()
NULL
> sessionInfo()
R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] tensorflow_2.11.0.9000 keras_2.11.0           reticulate_1.27-9000  

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10     here_1.0.1      lattice_0.20-45 png_0.1-8       withr_2.5.0     rprojroot_2.0.3
 [7] zeallot_0.1.0   rappdirs_0.3.3  grid_4.2.2      R6_2.5.1        jsonlite_1.8.4  magrittr_2.0.3 
[13] tfruns_1.5.1    rlang_1.0.6     cli_3.4.1       rstudioapi_0.14 whisker_0.4.1   Matrix_1.5-1   
[19] generics_0.1.3  tools_4.2.2     compiler_4.2.2  base64enc_0.1-3

Answer 1 · 2023-01-24T04:11:47.000Z

Hi, thanks for reporting.

From https://www.tensorflow.org/install/pip#windows-native:

Caution: TensorFlow 2.10 was the last TensorFlow release that supported GPU on native-Windows. Starting with TensorFlow 2.11, you will need to install TensorFlow in WSL2, or install tensorflow-cpu and, optionally, try the TensorFlow-DirectML-Plugin

It looks like WSL is your best bet if you want to continue to use Tensorflow with a GPU on Windows.

If you're using the RStudio IDE, here is a guide for how to setup the open-source rstudio server within WSL, so you can continue to use RStudio after moving your work to WSL: https://support.posit.co/hc/en-us/articles/360049776974-Using-RStudio-Server-in-Windows-WSL2

Answer 2 · 2023-01-24T05:01:16.000Z

Thanks so much for your reply, Tomasz.

Trying to limp along with 2.10 for a while before tackling WSL, but I've run into another problem. When I run ...

install.packages("keras")
library(reticulate)
virtualenv_create("r-reticulate", python = install_python())
library(keras)
install_keras(envname = "r-reticulate", version = "2.10")

... I still get ...

> tf_config()
TensorFlow v2.11.0 (C:\MINICO~1\lib\site-packages\tensorflow\__init__.p)
Python v3.9 (C:/Miniconda/python.exe)

... even after deleting keras and tensorflow from the library before installing. Any thoughts?

By the way, I love your book!

Answer 3 · 2023-01-24T14:24:54.000Z

The tf_config() is printing out a Windows short path, making it hard to diagnose what's going on exactly... but my best guess is that you have tensorflow installed in multiple locations, and reticulate is not picking the "right" python installation.

Some more steps to take:

Call normalizePath(tf_config()$location) to get the full path to the tensorflow installation being used. Note what conda environment it's in.
Call reticulate::py_discover_config() to get a printout of other python environments reticuate was able to find.
Give reticulate a directive about what python environment you want it to use. (You have to do this before reticulate has initialized Python). Either one of these two below should work:
- reticulate::use_condaenv("r-reticulate")
- Sys.setenv(RETICULATE_PYTHON = "C:/Path/to/miniconda/envs/r-reticulate/python.exe")

By the way, I love your book!

Thank you!!

Answer 4 · 2023-01-24T17:51:25.000Z

Tomasz, You were certainly right that I had multiple TF instances conflicting with one another. As I tried to remove them, it became like pulling on a loose thread until the entire sweater was a ball of yarn at my feet. In the end, I uninstalled R itself in hope of getting to something I could build on.

After re-installing reticulate, I ran the block I mentioned before ...

install.packages("keras")
library(reticulate)
virtualenv_create("r-reticulate", python = install_python())
library(keras)
install_keras(envname = "r-reticulate", version = "2.10")

That seemed to happen without a hitch...


Installation complete.


Restarting R session...

> library(tensorflow)
> tf_config()
No non-system installation of Python could be found.
Would you like to download and install Miniconda?
Miniconda is an open source environment management system for Python.
See https://docs.conda.io/en/latest/miniconda.html for more details.

Would you like to install Miniconda? [Y/n]: Y
* Installing Miniconda -- please wait a moment ...
* Downloading "https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe" ...
trying URL 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe'
Content type 'application/octet-stream' length 55508040 bytes (52.9 MB)
downloaded 52.9 MB

No non-system installation of Python could be found.
Would you like to download and install Miniconda?
Miniconda is an open source environment management system for Python.
See https://docs.conda.io/en/latest/miniconda.html for more details.

Would you like to install Miniconda? [Y/n]: Y
* Installing Miniconda -- please wait a moment ...
Error: miniconda installation failed [exit code 2]

To overcome Miniconda's issue with installing in a path with spaces, I created an environment variable to point to a conforming directory.

> miniconda_path()
[1] "C:\\r-miniconda"

Thanks for any help.

Answer 5 · 2023-01-24T21:50:05.000Z

Something wonky is definitely going on here, but I'm not quite sure what yet. Some things that stand out for me:

install_python() installs from python.org, and a virtual environment is created using just the standard library from python. conda or miniconda should not be in the mix here.
The fact that install_miniconda() raised an error due to a space in the path is surprising and likely a bug, but I think it's a separate issue.

Can you please copy and paste the full output from running these lines:

# install the development version of packages, in case the
# issue is already fixed but not on CRAN yet.
install.packages("remotes")
remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras")))

reticulate:::rm_all_reticulate_state() # start with a blank slate
library(reticulate)
miniconda_uninstall() # start with a blank slate
try(virtualenv_remove("r-tensorflow"))
virtualenv_create("r-tensorflow", python = install_python())
keras::install_keras(envname = "r-tensorflow", version = "2.10")

Answer 6 · 2023-01-24T22:12:10.000Z

Thanks so much for your reply, Tomasz.

Here's the full transcript:

> # install the development version of packages, in case the
> # issue is already fixed but not on CRAN yet.
> install.packages("remotes")
Installing package into ‘C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.2/remotes_2.4.2.zip'
Content type 'application/zip' length 399026 bytes (389 KB)
downloaded 389 KB

package ‘remotes’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
	C:\Users\Scott Lyden\AppData\Local\Temp\RtmpiMYra0\downloaded_packages
> remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras")))
Downloading GitHub repo rstudio/reticulate@HEAD
Running `R CMD build`...
* checking for file 'C:\Users\Scott Lyden\AppData\Local\Temp\RtmpiMYra0\remotes277c19115312\rstudio-reticulate-3bbc5b3/DESCRIPTION' ... OK
* preparing 'reticulate':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'reticulate_1.27-9000.tar.gz'
Installing package into ‘C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2’
(as ‘lib’ is unspecified)
* installing *source* package 'reticulate' ...
** using staged installation
** libs
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c event_loop.cpp -o event_loop.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c libpython.cpp -o libpython.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c output.cpp -o output.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c python.cpp -o python.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c readline.cpp -o readline.o
g++ -std=gnu++11  -I"C:/PROGRA~1/R/R-42~1.2/include" -DNDEBUG  -I'C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/Rcpp/include'   -I"C:/rtools42/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c signals.cpp -o signals.o
g++ -std=gnu++11 -shared -s -static-libgcc -o reticulate.dll tmp.def RcppExports.o event_loop.o libpython.o output.o python.o readline.o signals.o -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R/R-42~1.2/bin/x64 -lR
installing to C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2/00LOCK-reticulate/00new/reticulate/libs/x64
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (reticulate)
Downloading GitHub repo rstudio/tensorflow@HEAD
Running `R CMD build`...
* checking for file 'C:\Users\Scott Lyden\AppData\Local\Temp\RtmpiMYra0\remotes277c6f962bc6\rstudio-tensorflow-968af81/DESCRIPTION' ... OK
* preparing 'tensorflow':
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'tensorflow_2.11.0.9000.tar.gz'
Installing package into ‘C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2’
(as ‘lib’ is unspecified)
* installing *source* package 'tensorflow' ...
** using staged installation
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (tensorflow)
Downloading GitHub repo rstudio/keras@HEAD
Running `R CMD build`...
* checking for file 'C:\Users\Scott Lyden\AppData\Local\Temp\RtmpiMYra0\remotes277c2a4828cb\rstudio-keras-bc8ddf3/DESCRIPTION' ... OK
* preparing 'keras':
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
Removed empty directory 'keras/man-roxygen'
* building 'keras_2.11.0.9000.tar.gz'
Installing package into ‘C:/Users/Scott Lyden/AppData/Local/R/win-library/4.2’
(as ‘lib’ is unspecified)
* installing *source* package 'keras' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (keras)
> 
> reticulate:::rm_all_reticulate_state() # start with a blank slate
> library(reticulate)
> miniconda_uninstall() # start with a blank slate
> virtualenv_create("r-reticulate", python = install_python())
virtualenv: r-reticulate
> keras::install_keras(envname = "r-reticulate", version = "2.10")
Using virtual environment "r-reticulate" ...
Error: invalid version specification ‘’
In addition: Warning message:
In system2(python, c("-c", shQuote(command)), stdout = TRUE, stderr = TRUE) :
  running command '"C:/Users/Scott Lyden/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -c "import sys; import pip; sys.stdout.write(pip.__version__)"' had status 103
>

... and here's the traceback ...

> traceback()
8: stop(gettextf("invalid version specification %s", paste(sQuote(unique(x[!ok])), 
       collapse = ", ")), call. = FALSE, domain = NA)
7: .make_numeric_version(x, strict, .standard_regexps()$valid_numeric_version)
6: numeric_version(version)
5: pip_version(python)
4: virtualenv_install(envname = envname, packages = packages, ignore_installed = pip_ignore_installed, 
       ...)
3: reticulate::py_install(packages = packages, envname = envname, 
       method = method, conda = conda, python_version = python_version, 
       pip = TRUE, pip_ignore_installed = pip_ignore_installed, 
       ...)
2: tensorflow::install_tensorflow(method = method, conda = conda, 
       version = tensorflow, extra_packages = pkgs, pip_ignore_installed = pip_ignore_installed, 
       ...)
1: keras::install_keras(envname = "r-reticulate", version = "2.10")
>

Answer 7 · 2023-01-25T01:18:47.000Z

> virtualenv_create("r-reticulate", python = install_python())
virtualenv: r-reticulate

This output means that reticulate found an existing virtual env named "r-reticulate", and did nothing. If it was creating a new virtualenv, there would be much more output.

To go back to a clean slate, you can try virtualenv_remove("r-reticulate") or unlink("~/.virtualenvs/r-reticulate", recursive = TRUE). Note that on Windows, deleting a file can sometimes be surprisingly difficult. Windows will refuse to delete some files if another process has a file open, which can easily happen if an IDE (like Vs Code) had auto-magically activated the venv for you.

running command '"C:/Users/Scott Lyden/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -c "import sys; import pip; sys.stdout.write(pip.__version__)"' had status 103

This is a very vanilla python command that reticulate sends as a system call to get the pip version. This should not error, and the fact that it's erroring means that there is something fundamentally wrong with the installation. It could be any number of odd scenarios (e.g., a previous delete command was only partially successful).

If you open up cmd.exe and execute the same command, maybe you'll see a more fulsome error message that indicates what's wrong with the virtualenv. Or you can delete the virtualenv, create a fresh one, and see if that fixes the issue.

Answer 8 · 2023-01-25T04:25:55.000Z

Tomasz, I've gotten 99% of the way there (and I'm going to call that a win).

By toggling the existence of the r-reticulate virtual environment, I was able to see that for sure the virtual environment in question was located at C:/Users/Scott Lyden/Documents/.virtualenvs/r-reticulate, but when I ran tf_config() after these unsuccessful installation attempts, the python.exe in that directory tree was not listed among the python instances searched. Only python instances under c:\miniconda and c:\anaconda were being checked. After uninstalling Miniconda and Anaconda, I was able to successfully run install_keras().

Surprisingly, even though I ran ...

install.packages("keras")
library(reticulate)
virtualenv_create("r-reticulate", python = install_python())
library(keras)
install_keras(envname = "r-reticulate", version = "2.10")

... version 2.6.0 of Tensorflow was installed. In any case, TF is using the GPU and I'm back in business.

Thanks for all your help, Tomasz!

Answer 9 · 2023-01-25T13:41:44.000Z

reticulate echos the system commands it's issuing when doing python installation management actions. That is helpful for diagnosing issues exactly like this one. Look for the lines that start with a + prefix and note which python and pip is being invoked, and with what arguments.

If a different version of tensorflow is being installed than what was requested, it's possible that installation went through a different pathway than expected (e.g., through conda instead of pip).