Error Loading

Question

Error Loading

ilyakorsunsky opened this issue 6 years ago · 33 comments

Thanks for this implementation, really looking forward to having a native R/Rcpp implementation to use on my big datasets!

The package seems to install fine but then there is a problem loading the shared object. I am running this on R3.5. Do you know what this could be?

Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted
ERROR: loading failed

Answer 1 · 2018-07-01T19:59:37.000Z

I failed to correctly configure RcppParallel and was getting away with it on Windows when building the package via devtools.

I think I've fixed the underlying issue. Could you please try again with the latest build and let me know? I have added Appveyor and Travis CI to catch Linux (and Windows) installation issues.

What OS are you installing this on, by the way? Looks likes Mac?

Answer 2 · 2018-07-01T22:46:33.000Z

Tried it again on Mac and Linux (redhat?) and got the same error again

Answer 3 · 2018-07-01T23:45:41.000Z

~~Which versions of R are you running?~~ Sorry you, already said 3.5.

And do you know which version of Red Hat?

Answer 4 · 2018-07-02T00:15:46.000Z

It might also be helpful to know if the Mac version of R is the homebrew version or one installed from https://cran.r-project.org/bin/macosx/ -- and do any other packages that use C++ get loaded correctly?

Answer 5 · 2018-07-02T01:10:16.000Z

The Rcpp FAQ PDF http://dirk.eddelbuettel.com/code/rcpp/Rcpp-FAQ.pdf mentions "symbols not found" errors when registering dynamically loaded symbols and ensuring the RcppExports files are correctly regenerated.

I've updated the NAMESPACE entry to use useDynLib(uwot, .registration=TRUE) and regenerated the RcppExports. Maybe that will help? I am clutching at straws a bit due to being unable to reproduce this currently.

Answer 6 · 2018-07-02T09:32:02.000Z

Just realized that my redhat install is actually getting stuck on RcppParallel, so I'll need to fix that first

Answer 7 · 2018-08-02T05:29:58.000Z

Hi, I am having the same issue. I am on Mac (v 10.13.1) and R 3.3. My C++ packages appear to be loading correctly. Here is the error message

Thanks for your help.

==

Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
Referenced from: /Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/3.3/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted

Answer 8 · 2018-08-02T06:27:51.000Z

I'm sorry it's still not working for people on Macs. I've added OS X to the Travis CI configuration, but it's building correctly on there. I don't know if the output there is helpful.

Are you using the CRAN R or one from Homebrew? Is there any chance that other C++ libraries were built with clang but uwot is being forced to use gcc or something? I don't have access to a Mac so I can't do a lot more than google for similar error messages.

Answer 9 · 2018-08-02T07:08:41.000Z

Thanks for the prompt answer. I am using CRAN R, and as far as I can see clang is used. I tried the suggestions from Rccp FAQ pdf and removed R/RcppExports.R and src/RcppExports.cpp and then rebuild the package. Unfortunately, the error persists. Not sure what to do, since I know nothing of Rccp.

I'll update my R version do a clean install and see how that goes. I'll report the results.

Answer 10 · 2018-10-12T09:46:56.000Z

I noticed that this thread has not had any recent activity. Despite the fact that I have previously been able to install uwot, I have also experienced the problem described above and first noticed it ~2 weeks ago. Trying to install again today, I noted that it is still an issue. Since I was previously able to install the package I gradually stepped backwards in the commit history to find a "version" I can currently install. It turns out that although I can install 347f45d (Link to RcppParallel and RcppAnnoy., Jun 23), I cannot install 45fa4bd (Multi-threaded NN search., Jun 23). It would seem that either a) something in the later commit broke installation on some OSX or b) something has subsequently changed on some OSX since that change which has broken the installation.

I am using R version 3.5.1 and will include the full output at installation below. Hopefully this "clue" might help identify the issue so that it can be resolved.

Installing uwot
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
'/Users/jasonserviss/Github/uwot'
--library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'
--with-keep.source --install-tests

installing source package ‘uwot’ ...
** libs
make: Nothing to be done for `all'.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN3tbb4task13note_affinityEt
Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted
ERROR: loading failed
removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
Error: Command failed (1)

Answer 11 · 2018-10-12T16:25:41.000Z

Thank you for the detective work, @jasonserviss. Is this the error message from the latest build or from 45fa4bd? The only thing that jumps out at me from the diff between the two commits is that I was mixing std::size_t and size_t in the added function. I have fixed this inconsistency across the code base in the latest commit. Would be nice if that was the problem (probably not though)!

Previous reported errors are from symbols from functions in uwot, but __ZN3tbb4task13note_affinityEt is from the TBB library, which RcppParallel uses internally. This usually means that the NAMESPACE file hasn't correctly exported functions from RcppParallel, but we are doing that.

Otherwise it might mean that there is a problem with TBB. Seems hard to believe that RcppParallel could be correctly built and installed if TBB wasn't around, so I would expect to see that error if for some reason we aren't including TBB at link time. With g++ there should be a -ltbb argument during the link step. For example, the Appveyor CI Windows build has the following in 00install.out:

C:/Rtools/mingw_64/bin/g++ -shared -s -static-libgcc -o uwot.dll tmp.def RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -Lc:/RLibrary/RcppParallel/lib/x64 -ltbb -ltbbmalloc -Lc:/R/bin/x64 -lR

Unfortunately, I don't have this information in the travis Mac CI. The contents of 00install.out might be useful here, but I don't know how easy it is to get hold of that during a failed build. If anyone is using RStudio on Mac, they could try importing uwot as a project from the github repo and then the initial build output from devtools::load_all(".") would contain a lot more diagnostic info.

Answer 12 · 2018-10-12T18:12:44.000Z

It looks as though the error message was potentially from an earlier build (my fault, sorry for the confusion). The installation from both the most recent commit (sorry still not working) and the commit that worked (347f45d) are shown below. Below that I show devtools::load_all('.') for the freshly cloned repo (I print the DESCRIPTION file to show the version). Not sure any of this is going to help a lot... Let me know if there is anything else I can do that comes to mind.

devtools::install_github('jlmelville/uwot', ref = 'def7898e7738f63f5be5ce9aa811baad4f6909b7')
#> Downloading GitHub repo jlmelville/uwot@def7898e7738f63f5be5ce9aa811baad4f6909b7
#> from URL https://api.github.com/repos/jlmelville/uwot/zipball/def7898e7738f63f5be5ce9aa811baad4f6909b7
#> Installing uwot
#> '/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
#>   --no-environ --no-save --no-restore --quiet CMD INSTALL  \
#>   '/private/var/folders/6w/_8bn209n7_dbqbk7hdst6hfm0000gp/T/RtmpFkwl6D/devtoolsd9c14db71ce9/jlmelville-uwot-def7898'  \
#>   --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
#>   --install-tests
#> 
#> Installation failed: Command failed (1)
  devtools::install_github('jlmelville/uwot', ref = '347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a')
#> Downloading GitHub repo jlmelville/uwot@347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a
#> from URL https://api.github.com/repos/jlmelville/uwot/zipball/347f45d4ed1fd0c9f4377bb10b0acf36c9ac2c7a
#> Installing uwot
#> '/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
#>   --no-environ --no-save --no-restore --quiet CMD INSTALL  \
#>   '/private/var/folders/6w/_8bn209n7_dbqbk7hdst6hfm0000gp/T/RtmpFkwl6D/devtoolsd9c172a7b431/jlmelville-uwot-347f45d'  \
#>   --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
#>   --install-tests
#>

standard output and standard error

* installing *source* package ‘uwot’ ...
** libs
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c nn_parallel.cpp -o nn_parallel.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c supervised.cpp -o supervised.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c transform.cpp -o transform.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o uwot.so RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: package or namespace load failed for ‘uwot’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so':
  dlopen(/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
  Referenced from: /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
  Expected in: flat namespace
 in /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs/uwot.so
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot’

* installing *source* package ‘uwot’ ...
** libs
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
/usr/local/clang4/bin/clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/clang4/lib -o uwot.so RcppExports.o gradient.o optimize.o perplexity.o smooth_knn.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (uwot)

Created on 2018-10-12 by the reprex package (v0.2.0).

Session info

devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  tz       Europe/Stockholm            
#>  date     2018-10-12
#> Packages -----------------------------------------------------------------
#>  package   * version date       source        
#>  backports   1.1.2   2017-12-13 CRAN (R 3.5.0)
#>  base      * 3.5.1   2018-07-05 local         
#>  compiler    3.5.1   2018-07-05 local         
#>  curl        3.2     2018-03-28 CRAN (R 3.5.0)
#>  datasets  * 3.5.1   2018-07-05 local         
#>  devtools    1.13.6  2018-06-27 CRAN (R 3.5.0)
#>  digest      0.6.17  2018-09-12 CRAN (R 3.5.1)
#>  evaluate    0.11    2018-07-17 CRAN (R 3.5.0)
#>  git2r       0.23.0  2018-07-17 CRAN (R 3.5.0)
#>  graphics  * 3.5.1   2018-07-05 local         
#>  grDevices * 3.5.1   2018-07-05 local         
#>  htmltools   0.3.6   2017-04-28 CRAN (R 3.5.0)
#>  httr        1.3.1   2017-08-20 CRAN (R 3.5.0)
#>  knitr       1.20    2018-02-20 CRAN (R 3.5.0)
#>  magrittr    1.5     2014-11-22 CRAN (R 3.5.0)
#>  memoise     1.1.0   2017-04-21 CRAN (R 3.5.0)
#>  methods   * 3.5.1   2018-07-05 local         
#>  R6          2.2.2   2017-06-17 CRAN (R 3.5.0)
#>  Rcpp        0.12.19 2018-10-01 CRAN (R 3.5.0)
#>  rmarkdown   1.10    2018-06-11 cran (@1.10)  
#>  rprojroot   1.3-2   2018-01-03 CRAN (R 3.5.0)
#>  stats     * 3.5.1   2018-07-05 local         
#>  stringi     1.2.4   2018-07-20 CRAN (R 3.5.0)
#>  stringr     1.3.1   2018-05-10 CRAN (R 3.5.0)
#>  tools       3.5.1   2018-07-05 local         
#>  utils     * 3.5.1   2018-07-05 local         
#>  withr       2.1.2   2018-03-15 CRAN (R 3.5.0)
#>  yaml        2.2.0   2018-07-25 CRAN (R 3.5.0)

setwd('~/Desktop/uwot-master/')
readLines('DESCRIPTION')
#>  [1] "Package: uwot"                                                                                               
#>  [2] "Title: The Uniform Manifold Approximation and Projection (UMAP) Method for Dimensionality Reduction"         
#>  [3] "Version: 0.0.0.9004"                                                                                         
#>  [4] "Authors@R: person(\"James\", \"Melville\", email = \"james.melville@gmail.com\", role = c(\"aut\", \"cre\"))"
#>  [5] "Author: James Melville <james.melville@gmail.com>"                                                           
#>  [6] "Maintainer: James Melville <james.melville@gmail.com>"                                                       
#>  [7] "Description: The dimensionality reduction method UMAP "                                                      
#>  [8] "  (McInnes and Healy, 2018, <arxiv:1802.03426>). It is similar "                                             
#>  [9] "  to, but faster than, the t-Distributed Stochastic Neighbor Embedding (t-SNE) "                             
#> [10] "  method."                                                                                                   
#> [11] "License: GPL-3"                                                                                              
#> [12] "URL: https://github.com/jlmelville/uwot"                                                                     
#> [13] "BugReports: https://github.com/jlmelville/uwot/issues"                                                       
#> [14] "Encoding: UTF-8"                                                                                             
#> [15] "LazyData: true"                                                                                              
#> [16] "Suggests: testthat,"                                                                                         
#> [17] "    covr"                                                                                                    
#> [18] "RoxygenNote: 6.1.0"                                                                                          
#> [19] "Depends: Matrix"                                                                                             
#> [20] "LinkingTo: Rcpp, RcppArmadillo, RcppProgress, RcppParallel, RcppAnnoy"                                       
#> [21] "Imports: Rcpp,"                                                                                              
#> [22] "    methods,"                                                                                                
#> [23] "    FNN,"                                                                                                    
#> [24] "    RSpectra,"                                                                                               
#> [25] "    RcppAnnoy,"                                                                                              
#> [26] "    RcppParallel,"                                                                                           
#> [27] "    irlba"                                                                                                   
#> [28] "SystemRequirements: GNU make"
devtools::load_all('.')
#> Loading uwot
#> Loading required package: Matrix
#> Error in dyn.load(dllfile): unable to load shared object '/Users/jasonserviss/Desktop/uwot-master/src/uwot.so':
#>   dlopen(/Users/jasonserviss/Desktop/uwot-master/src/uwot.so, 6): Symbol not found: __ZN13umap_gradient8clip_maxE
#>   Referenced from: /Users/jasonserviss/Desktop/uwot-master/src/uwot.so
#>   Expected in: flat namespace
#>  in /Users/jasonserviss/Desktop/uwot-master/src/uwot.so

Created on 2018-10-12 by the reprex package (v0.2.0).

Answer 13 · 2018-10-13T06:47:02.000Z

Thanks for the detailed output. The good news is that at least we are back to having a consistent error message.

The symbol that can't be found is clip_max, which is a static constexpr in the gradient classes, but which is accessed via a member instance. I can't see why that isn't ok, but to simplify matters I have converted it to a member variable which is initialized in the constructor initializer list. I also discovered that I had neglected to add any header guards in the header files, which I have also remedied.

At this point, I doubt if this will fix the problem, but it might change the error message to something more useful.

Some other things to check on: are you using a custom ~/.R/Makevars file? There's a possibility this is some g++ vs clang issue, but I'm not sure how the Mac package binaries are built or whether it matters.

One other thing you could try is installing the package and tell it not to delete uwot.so when it fails:

devtools::install_github("jlmelville/uwot", args="--no-clean-on-error")

Then try running otool -L /Users/jasonserviss/Desktop/uwot-master/src/uwot.so and at least make sure everything is linking to where it ought to be.

Answer 14 · 2018-10-13T07:05:21.000Z

Yes, I have a custom Makevars. It contains:

LDFLAGS= -L/usr/local/clang4/lib
CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++
CXX1X=/usr/local/clang4/bin/clang++
CXX98=/usr/local/clang4/bin/clang++
CXX11=/usr/local/clang4/bin/clang++
CXX14=/usr/local/clang4/bin/clang++
CXX17=/usr/local/clang4/bin/clang++

I ended up with this custom Makevars by enabling OpenMP use in R via: OpenMP in R on OS X. If I temporarily remove the custom Makevars, I can install version 0.0.0.9004 (e34ae33) without any issues. So I guess that might isolate the "problem". I am not so familiar with different compilers to really be able to understand the details of what is going on (you indicate in your comment "some g++ vs clang issue"). Maybe it is possible to setup your Travis build to use clang to help you further diagnose the problem...? Or, alternatively, if I can do anything else on my end let me know.

Output below:

Installing uwot
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ  \
  --no-save --no-restore --quiet CMD INSTALL  \
  '/Users/jasonserviss/Desktop/uwotCleanNew/uwot-master'  \
  --library='/Library/Frameworks/R.framework/Versions/3.5/Resources/library'  \
  --with-keep.source --install-tests 

* installing *source* package ‘uwot’ ...
** libs
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c RcppExports.cpp -o RcppExports.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c gradient.cpp -o gradient.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c nn_parallel.cpp -o nn_parallel.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c optimize.cpp -o optimize.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c perplexity.cpp -o perplexity.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c smooth_knn.cpp -o smooth_knn.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c supervised.cpp -o supervised.o
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DARMA_64BIT_WORD=1 -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppArmadillo/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppProgress/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppParallel/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/RcppAnnoy/include" -I/usr/local/include   -fPIC  -Wall -g -O2 -c transform.cpp -o transform.o
clang++ -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o uwot.so RcppExports.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/uwot/libs
** R
** tests
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (uwot)

Answer 15 · 2018-10-13T19:15:13.000Z

The problem might not be clang in general, but the version of clang that you're using. I currently know nothing about the various details of what compiler toolchain is used to build the R package binaries on Mac, what versions of which compilers are compatible with each other, what linker flags are needed and so on. I'm sure a definitive answer lies in various FAQs and the combined wisdom of googling for the "symbol not found" error with respect to other Rcpp-using projects, but not being a Mac user, I may not be the right person for this particular job.

For now I am suggesting not using a custom Makevars during building. I would be keen to hear if this works (or doesn't) for others.

Answer 16 · 2018-12-12T19:16:10.000Z

I have the same issue on my mac!

Answer 17 · 2018-12-12T19:52:36.000Z

I did a clean install of R 3.5.1. and the package now installs and I can run the examples. Hope this helps. My OS is 10.13.1

Answer 18 · 2018-12-13T04:02:33.000Z

@robsalasco, are you using a custom Makevars?

@good-marketing, glad you got it fixed, were you also using a custom Makevars? With the clean install of R, was the library also created fresh? Perhaps installing the newest versions of Rcpp and RcppParallel can help.

Answer 19 · 2018-12-13T14:18:15.000Z

@robsalasco, are you using a custom Makevars?

Hi @jlmelville! I tried compiling with and without custom Makevars.

Answer 20 · 2018-12-13T17:25:22.000Z

The good news is that at least I was able to reproduce this error by attempting to build on r-hub with the "macOS 10.9 Mavericks, R-oldrel" configuration.

The bad news is I don't know what's causing the problem.

Answer 21 · 2018-12-14T05:05:32.000Z

I would guess that it's because optimize.cpp refers to gradient.clip_max but it should really be Gradient::clip_max now that clip_max is a static data member.

Answer 22 · 2018-12-14T07:10:22.000Z

I don't know why gradient.clip_max causes problems: has there been a change to the C++ standard that disallows static member access through an instance? Thanks to @LTLA anyway.

The r-hub "macOS 10.9 Mavericks, R-oldrel" configuration that was failing is now passing again. This gives me hope that we are back to the situation where only custom Makevars is causing persistent problems.@robsalasco, if you are able, would you attempting to re-install from the current master?

For anyone else with problems, here's an example of the successful r-hub compilation output. Perhaps someone can compare with the flags and paths they are using and spot any potential differences (this is for R 3.3.3):

2723#> clang++ -std=c++11 -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -DARMA_64BIT_WORD=1 -I/usr/local/include -I/usr/local/include/freetype2 -I/opt/X11/include -I"/Users/usergUphdZt8/R/Rcpp/include" -I"/Users/usergUphdZt8/R/RcppArmadillo/include" -I"/Users/usergUphdZt8/R/RcppProgress/include" -I"/Users/usergUphdZt8/R/RcppParallel/include" -I"/Users/usergUphdZt8/R/RcppAnnoy/include" -fPIC -Wall -mtune=core2 -g -O2 -c transform.cpp -o transform.o

2724#> clang++ -std=c++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o uwot.so RcppExports.o connected_components.o gradient.o nn_parallel.o optimize.o perplexity.o smooth_knn.o supervised.o transform.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation

Answer 23 · 2018-12-14T07:22:44.000Z

Possibly something esoteric to do with constexpr? TBH I've never tried to access static members via an instance, mostly for semantic reasons.

Edit: see the end of https://en.cppreference.com/w/cpp/language/static, which suggests that we needed a definition in gradient.cpp along the lines of:

constexpr int umap_gradient::clip_max;
constexpr int apumap_gradient::clip_max;
constexpr int tumap_gradient::clip_max;
constexpr int largevis_gradient::clip_max;

Answer 24 · 2018-12-15T05:28:30.000Z

@jlmelville 🎉 it works! the current master compiles without problem :)

Answer 25 · 2018-12-15T06:07:55.000Z

Thanks for the feedback @robsalasco. Hopefully we are back to normal.

Answer 26 · 2018-12-15T12:21:12.000Z

@jlmelville thanks for the fix! hope the package can be submitted to CRAN soon.

Answer 27 · 2019-04-13T02:22:06.000Z

uwot is now on CRAN. Hopefully the Mac binaries can be used without problems. I will eventually close this issue if there is no re-occurrence.

Answer 28 · 2019-04-16T09:21:47.000Z

CRAN version works for me (although I haven't been using a custom Makevars for sometime).

Answer 29 · 2020-10-13T21:00:07.000Z

Hi! I have just installed the CRAN version and most recent Github version and both return the same error when trying to use the umap function:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so':
  dlopen(/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so, 6): Symbol not found: _EXTPTR_PTR
  Referenced from: /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
  Expected in: /Library/Frameworks/R.framework/Resources/lib/libR.dylib
 in /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so

Here is my sessionInfo():

R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.7

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] uwot_0.1.8.9001 Matrix_1.2-18   word2vec_0.3.2  udpipe_0.8.4-1  ggplot2_3.3.2   lares_4.9.5     dplyr_1.0.2    
[8] openxlsx_4.2.2  h2o_3.30.1.3   

loaded via a namespace (and not attached):
 [1] tinytex_0.25      tidyselect_1.1.0  xfun_0.16         purrr_0.3.4       lattice_0.20-41   colorspace_1.4-1 
 [7] vctrs_0.3.4       generics_0.0.2    htmltools_0.5.0   yaml_2.2.1        rlang_0.4.8       pillar_1.4.6     
[13] glue_1.4.2        withr_2.3.0       lifecycle_0.2.0   plyr_1.8.6        stringr_1.4.0     munsell_0.5.0    
[19] gtable_0.3.0      rvest_0.3.6       zip_2.1.1         evaluate_0.14     knitr_1.29        Rcpp_1.0.5       
[25] scales_1.1.1      jsonlite_1.7.1    config_0.3        packrat_0.5.0     digest_0.6.25     stringi_1.5.3    
[31] rlist_0.4.6.1     grid_4.0.0        tools_4.0.0       bitops_1.0-6      magrittr_1.5      RCurl_1.98-1.2   
[37] patchwork_1.0.1   tibble_3.0.3      crayon_1.3.4      tidyr_1.1.2       pkgconfig_2.0.3   ellipsis_0.3.1   
[43] data.table_1.13.0 xml2_1.3.2        pROC_1.16.2       lubridate_1.7.9   rmarkdown_2.3     httr_1.4.2       
[49] rstudioapi_0.11   R6_2.4.1          compiler_4.0.0

Answer 30 · 2020-10-14T14:39:03.000Z

hi @laresbernardo is your install of RcppAnnoy ok? Does library(RcppAnnoy) work, for instance?

Answer 31 · 2020-10-14T15:00:39.000Z

hi @laresbernardo is your install of RcppAnnoy ok? Does library(RcppAnnoy) work, for instance?

Hi @jlmelville Thanks for reaching back!
No, I get a similar error when calling the library:

> library(RcppAnnoy)
Error: package or namespace load failed for ‘RcppAnnoy’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so':
  dlopen(/Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so, 6): Symbol not found: _EXTPTR_PTR
  Referenced from: /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
  Expected in: /Library/Frameworks/R.framework/Resources/lib/libR.dylib
 in /Users/bernardolares/Library/R/4.0/library/RcppAnnoy/libs/RcppAnnoy.so
In addition: Warning message:
package ‘RcppAnnoy’ was built under R version 4.0.2

Any idea as of why is this happening and/our how to fix it?

Answer 32 · 2020-10-14T15:31:31.000Z

@laresbernardo this seems to be a problem with installing RcppAnnoy, not a uwot problem so I can only give typical advice about package problems: try removing RcppAnnoy and then reinstalling it. If you are able to, see if upgrading to R 4.0.2 helps. See if CRAN vs installing from the RcppAnnoy github repo makes a difference. Sorry I can't be more help.

Answer 33 · 2021-08-06T17:51:57.000Z

It seems appropriate to close this after a long period of inactivity (thus guaranteeing it will start causing problems again, but I'll take the risk).