MPI.jl fails to find homebrew MPI in macos Sonoma runner

Question

MPI.jl fails to find homebrew MPI in macos Sonoma runner

miguelbiron opened this issue a year ago · 15 comments

miguelbiron commented a year ago

Hi folks -- as the title says, we're getting the following error in our CI workflow

https://github.com/Julia-Tempering/Pigeons.jl/actions/runs/8849574516/job/24301884110#step:7:1236

The same happens for openmpi. Not sure if this is an issue on your end or upstream (julia's Libdl).

Answer 1 · 2024-04-26T15:03:40.000Z

Why not just use the default MPI?

Answer 2 · 2024-04-26T15:07:45.000Z

Because at some point we did just that, and we realized that many of our users were finding issues that were not being picked up by our CI tests.

Answer 3 · 2024-04-26T15:09:50.000Z

BTW we basically copied the workflow from you, so we expect that if you rerun your CI you should see the same issues (your last run was 3 days ago and used the old runners).

Answer 4 · 2024-04-26T15:59:03.000Z

Is this because macos-latest now is aarch64-darwin?

Answer 5 · 2024-04-26T16:01:12.000Z

Is this because macos-latest now is aarch64-darwin?

I don't know if that's the cause, but yes, they are arm now

Answer 6 · 2024-04-26T16:09:04.000Z

Would setting macos-13 instead of macos-latest help?

Answer 7 · 2024-04-26T16:25:37.000Z

That or macos-12 worked for us (on a similar issue, but unrelated to MPI.jl)

Answer 8 · 2024-04-26T16:50:08.000Z

Testing this right now Julia-Tempering/Pigeons.jl#230. Went with macos-12 because this was used in the last successful run

Answer 9 · 2024-04-26T16:58:28.000Z

We probably still want to understand what's happening, but currently I personally don't have the time to do that, but at least using an older macos image should help with getting a working environment.

Answer 10 · 2024-04-26T17:01:51.000Z

Ok confirmed that using macos-12 still works. Per @giordano's comment I'll leave the issue open until a more permanent solution arises.

Answer 11 · 2024-04-27T03:49:07.000Z

It's failing here as well:
https://github.com/JuliaParallel/MPI.jl/actions/runs/8807615431/job/24305646282

I don't really have the time to look into it at the moment though.

Answer 12 · 2024-04-29T16:54:31.000Z

Ah, I think I understand now: up to the macos-13 image, the brew commands used to install packages in /usr/local (which is probably a default search path for the dynamic loader), but in macos-14 they're now installed under /opt/homebrew. I don't know whether this is a change in Homebrew, or macOS, or whatever, but the net result is that libmpi isn't anymore in a default search path for the dynamic loader, and so we have to do something #831 to reliably make loading libmpi. An alternative could be to tell Homebrew to install the packages again in /usr/local, but I think setting DYLD_FALLBACK_LIBRARY_PATH is slightly more general: we don't have to change the installation directory.

Originally posted by @giordano in #829 (comment)

Answer 13 · 2024-04-29T17:31:38.000Z

Thanks for fixing this! I'll borrow your solution for our own MPI tests.

Answer 14 · 2024-04-29T17:35:36.000Z

I think #832 is slightly nicer as it doesn't involve messing with environment variables (but that requires a change in MPIPreferences.jl).

Answer 15 · 2024-04-29T17:44:41.000Z

Ah thanks yes I just saw that. Will switch to that approach once it is merged