Tests fail on Mac M1+

Closed this issue a year ago · 15 comments

── Failed tests ──────────────────────────────────
Failure (test-fit.R:300:3): refit_multiple_optimizers works with parallel computations and selected optimizers
logLik(result) > logLik(fit) is not TRUE

`actual`:   FALSE
`expected`: TRUE

Session Info:

R version 4.3.1 (2023-06-16)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.4

Apple clang version 14.0.3 (clang-1403.0.22.14.1)
MacOSX13.3.sdk

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: Europe/Zurich
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mmrm_0.2.2.9032

loaded via a namespace (and not attached):
 [1] backports_1.4.1   Matrix_1.6-0      lattice_0.21-8    magrittr_2.0.3    glue_1.6.2       
 [6] stringr_1.5.0     parallel_4.3.1    lifecycle_1.0.3   Rdpack_2.4        cli_3.6.1        
[11] parallelly_1.36.0 grid_4.3.1        compiler_4.3.1    rbibutils_2.2.13  rstudioapi_0.15.0
[16] tools_4.3.1       checkmate_2.2.0   nlme_3.1-162      Rcpp_1.0.11       rlang_1.1.1      
[21] stringi_1.7.12

danielinteractive commented a year ago

OK this could be due to https://stackoverflow.com/questions/70638118/configuring-compilers-on-apple-silicon-big-sur-monterey-ventura-for-rcpp-and
I guess I need to set up my toolchain as recommended, since now tons of other errors 🆘

danielinteractive commented a year ago

Trying also to follow tips at https://stat.ethz.ch/R-manual/R-devel/doc/manual/R-admin.html#Building-binary-packages

danielinteractive commented a year ago

Also reinstalled TMB and glmmTMB from source and ran their tests and they worked fine without errors

danielinteractive commented a year ago

Can maybe try https://mac.r-project.org/macbuilder/submit.html next and see if that binary would pass the tests

danielinteractive commented a year ago

Running:

MacBuilder: https://mac.r-project.org/macbuilder/results/1690270381-ee47bbb118ee3b0d/
- This also leads to test failures:
R-Hub compiled code tests: https://builder.r-hub.io/status/mmrm_0.2.2.9032.tar.gz-b6bd40e81d8b812f2600fa58cde43d57 (does not have Mac systems in options)
- This does not work for us because many dependencies are (for some reason) not available (even though they are on CRAN)

danielinteractive commented a year ago

Next we need to:

find out where the tests broke for M1+
- go back in the version history on main branch and test:
  - Tried locally with Install, Test (don't click Check because then it recompiles the vignettes which takes forever)
  - Problem: does not work locally because even 0.2.2 compiled locally fails the tests. But with CRAN 0.2.2 the tests pass. (This is a separate issue on my local toolchain apparently)
- bisection search for the commit where it breaks but previous commit passes tests
  - 9033: breaks
  - 9016: breaks
  - 9008: breaks
  - 9004: breaks
  - 0.2.2: also breaks!!
- build corresponding source package manually since no longer on R-universe or elsewhere:
  - R CMD build --no-build-vignettes mmrm
- submit to Mac builder
understand what changes in C++ code are suspicious for breaking the tests
fix the problem by undoing these changes

danielinteractive commented a year ago

So the problem is that the Mac Builder does not help us sufficiently here, because even the current CRAN release (where tests based on CRAN binary pass fine locally and on CRAN) leads to test failures on Mac Builder. Why is that?

Differences:

CRAN:
- R was compiled by
  - Apple clang version 14.0.0 (clang-1400.0.29.202)
  - GNU Fortran (GCC) 12.2.0
- running under: macOS Big Sur 11.6.7
- used C++ compiler: ‘Apple clang version 13.0.0 (clang-1300.0.29.30)’ (!)
- used SDK: ‘MacOSX11.3.sdk’
Mac Builder:
- R was compiled by (same)
  - Apple clang version 14.0.0 (clang-1400.0.29.202)
  - GNU Fortran (GCC) 12.2.0
- running under: macOS Ventura 13.3.1 (different)
- used C++ compiler: ‘Apple clang version 14.0.3 (clang-1403.0.22.14.1)’ (different)
- used SDK: ‘MacOSX11.3.sdk’ (same)

danielinteractive commented a year ago

Sent an email to https://stat.ethz.ch/mailman/listinfo/r-sig-mac: https://stat.ethz.ch/pipermail/r-sig-mac/2023-July/014837.html

danielinteractive commented a year ago

Next step could be to try a lower version of clang (thanks @gowerc)

brew install llvm@13
- To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/opt/homebrew/opt/llvm@13/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@13/lib/c++"
- llvm@13 is keg-only, which means it was not symlinked into /opt/homebrew because this is an alternate version of another formula.
- Using that for now: If you need to have llvm@13 first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/llvm@13/bin:$PATH"' >> ~/.zshrc
  just to see if it changes anything
- For compilers to find llvm@13 you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/llvm@13/lib" and export CPPFLAGS="-I/opt/homebrew/opt/llvm@13/include"
Installed brew
- Read the analytics documentation (and how to opt-out) here:
  ⁃ https://docs.brew.sh/Analytics
  ⁃ To add Homebrew to your PATH run brew shellenv in your shell profile (e.g. ~/.bash_profile or ~/.zprofile)
  ⁃ Run brew help to get started
  ⁃ Further documentation:
  ⁃ https://docs.brew.sh/
be careful though to not add it to your PATH as you don't really want to be using a different compiler to what most of your software was compiled by so would just have a project specific Makevars file to override the compiler for that project

danielinteractive commented a year ago

Other next steps:

Try development toolchain on Macbuilder: https://mac.r-project.org/macbuilder/results/1692024092-ba6e80bfa6b55bc7/
Install additional CLT versions from https://developer.apple.com/download/all/

danielinteractive commented a year ago

@hrbrmstr Hi Bob, just got the tip from @yonicd that you might have some pointers about these kinds of issues :-) Any ideas?

danielinteractive commented a year ago

Today tried to install CLT 13.4 but it does not work because the Mac is too new:

danielinteractive commented a year ago

Development toolchain on Mac Builder also uses 14.0.3 clang:

and (therefore?) still gives test failures:

danielinteractive commented a year ago

OK so with the following I can get the current main branch tests to pass completely*:

Install llvm@13 (see above) with brew

In .R/Makevars:

CC=/opt/homebrew/opt/llvm@13/bin/clang
CXX=/opt/homebrew/opt/llvm@13/bin/clang++
CXX14=/opt/homebrew/opt/llvm@13/bin/clang++
LDFLAGS += "-L/opt/homebrew/opt/llvm@13/lib"
CPPFLAGS += "-I/opt/homebrew/opt/llvm@13/include"

This means that if build and check the package, then I am using the clang 13 version consistently.
No need to update .zshrc so other than R packages nothing should be changed, and the single Makevars file could be just deactivated easily.

* OK there is a minor warning about a symbol ‘__ZNSt3__14coutE’ being found in tmb.o but I don't worry about this for now

danielinteractive commented a year ago

closing for now since we have a workaround