How to install with existing juliaup?
Closed this issue · 8 comments
I tried to install juhpc on a system where I already have Julia (and juliaup) installed. juhpc fails with the error messages
[ JUHPC: Starting installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: Installing temporary juliaup installation in /dev/shm/eschnett/juliaup...
hw.optional.arm64: 1
info: downloading installer
Welcome to Julia!
It seems that Juliaup is already installed on this system. Please remove the previous installation of Juliaup before you try to install a new version.
[ JUHPC: ... done.
[ JUHPC: Creating preferences for HPC key packages...
Error: Failed to run `run_command_add` from the `run_command_initial_setup_from_launcher` command.
Caused by:
0: Failed to update versions db.
1: `run_command_update_version_db` command failed to load configuration db.
2: Could not create juliaup home folder.
3: Operation not permitted (os error 1)
Error: The Julia launcher failed to load a configuration file.
Caused by:
0: Could not create juliaup home folder.
1: Operation not permitted (os error 1)
[ JUHPC: ... done.
[ JUHPC: Creating wrapper for juliaup...
Error: Failed to run `run_command_add` from the `run_command_initial_setup_from_launcher` command.
Caused by:
0: Failed to update versions db.
1: `run_command_update_version_db` command failed to load configuration db.
2: Could not create juliaup home folder.
3: Operation not permitted (os error 1)
Error: The Julia launcher failed to load a configuration file.
Caused by:
0: Could not create juliaup home folder.
1: Operation not permitted (os error 1)
[ JUHPC: ... done.
[ JUHPC: Creating activation script...
Error: Failed to run `run_command_add` from the `run_command_initial_setup_from_launcher` command.
Caused by:
0: Failed to update versions db.
1: `run_command_update_version_db` command failed to load configuration db.
2: Could not create juliaup home folder.
3: Operation not permitted (os error 1)
Error: The Julia launcher failed to load a configuration file.
Caused by:
0: Could not create juliaup home folder.
1: Operation not permitted (os error 1)
[ JUHPC: ... done.
[ JUHPC: cleaning up temporary juliaup installation in /dev/shm/eschnett/juliaup.
[ JUHPC: ... Installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries is complete.
I think this might be a common scenario. People wouldn't like to have to tear down their current Julia setup just to give juhpc a try.
How to I skip installing juliaup?
Good point! This should be resolved in #14 (already merged).
Can you please retry setting VERSION=main?
I did exactly this. I did not delete any existing files/directories that were created during the failed attempt earlier.
There were no error messages.
The file $HOME/juhpc_setup/activate is empty.
The next step in the example, running juliaup without arguments, then fails, because this runs the original juliaup which doesn't work with arguments.
When I rerun after deleting $HOME/juhpc_setup, then I see this output:
$ /bin/bash -l /tmp/juhpc /Users/eschnett/juhpc_setup /Users/eschnett/juliaup
/Users/eschnett/.bash_profile: line 84: /Users/eschnett/miniforge3/etc/profile.d/conda.sh: No such file or directory
\e[1;35m ██╗\e[0m \e[1;32m██╗ ██╗\e[0m \e[1;31m██╗ ██╗██████╗ ██████╗\e[0m
\e[1;35m ██║\e[0m \e[1;32m██║ ██║\e[0m \e[1;31m██║ ██║██╔══██╗██╔════╝\e[0m
\e[1;35m ██║\e[0m \e[1;32m██║ ██║\e[0m \e[1;31m███████║██████╔╝██║ \e[0m
\e[1;35m██╗ ██║\e[0m \e[1;32m██║ ██║\e[0m \e[1;31m██╔══██║██╔═══╝ ██║ \e[0m
\e[1;35m╚█████╔╝\e[0m \e[1;32m╚██████╔╝\e[0m \e[1;31m██║ ██║██║ ╚██████╗\e[0m
\e[1;35m ╚════╝ \e[0m \e[1;32m ╚═════╝ \e[0m \e[1;31m╚═╝ ╚═╝╚═╝ ╚═════╝\e[0m
\e[1;34m[\e[0m \e[1;35mJ\e[0m\e[1;32mU\e[0m\e[1;31mH\e[0m\e[1;31mP\e[0m\e[1;31mC\e[0m\e[1;34m:\e[0m Starting installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries...
\e[1;34m[\e[0m \e[1;35mJ\e[0m\e[1;32mU\e[0m\e[1;31mH\e[0m\e[1;31mP\e[0m\e[1;31mC\e[0m\e[1;34m:\e[0m Installing temporary juliaup installation in /dev/shm/eschnett/juliaup...
hw.optional.arm64: 1
info: downloading installer
Welcome to Julia!
This will download and install the official Julia Language distribution
and its version manager Juliaup.
Juliaup will be installed into the Juliaup home directory, located at:
/dev/shm/eschnett/juliaup
The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:
/dev/shm/eschnett/juliaup/bin
You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.
Now installing Juliaup
Error: Failed to create install folder for Juliaup.
Caused by:
Operation not permitted (os error 1)
\e[1;34m[\e[0m \e[1;35mJ\e[0m\e[1;32mU\e[0m\e[1;31mH\e[0m\e[1;31mP\e[0m\e[1;31mC\e[0m\e[1;34m:\e[0m \e[1;31mERROR:\e[0m failed to install Juliaup (and Julia) in /dev/shm/eschnett/juliaup.
\e[1;34m[\e[0m \e[1;35mJ\e[0m\e[1;32mU\e[0m\e[1;31mH\e[0m\e[1;31mP\e[0m\e[1;31mC\e[0m\e[1;34m:\e[0m cleaning up temporary juliaup installation in /dev/shm/eschnett/juliaup.
There were no error messages.
I added an error when the temporal installation fails.
The file $HOME/juhpc_setup/activate is empty.
Given that the temporal installation failed everything else failed. Should you run again in the same situation then it should error immediately and abort.
I did another small improvement and I just tried myself to install a setup using JUHPC, having already a pre existing juliaup installation, with entries in .bashrc and .profile. You can see below after a fresh login, that first the pre existing installation is active and then the new installation becomes active when I source the activate script:
[todi][omlins@todi-ln002 ~]$ julia --version
The latest version of Julia in the `release` channel is 1.10.5+0.aarch64.linux.gnu. You currently have `1.10.4+0.aarch64.linux.gnu` installed. Run:
juliaup update
in your terminal shell to install Julia 1.10.5+0.aarch64.linux.gnu and update the `release` channel to that version.
julia version 1.10.4
[todi][omlins@todi-ln002 ~]$ JUHPC_SETUP_INSTALLDIR=$SCRATCH/${HOSTNAME%%-*}/juhpc_setup # HPC setup installation environment variables must be expanded during installation.
[todi][omlins@todi-ln002 ~]$ . $JUHPC_SETUP_INSTALLDIR/activate
[ JUHPC: Activating HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: PATH: /capstor/scratch/cscs/omlins/todi/juhpc_setup/juliaup_wrapper:/capstor/scratch/cscs/omlins/todi/juliaup/bin:/capstor/scratch/cscs/omlins/julia_local/bin:/capstor/scratch/cscs/omlins/julia_local/julia_latest/bin/:/users/omlins/stackinator/bin/:/usr/local/bin:/usr/bin:/bin:/usr/lpp/mmfs/bin:/usr/lib/mit/bin
[ JUHPC: JULIAUP_DEPOT_PATH: /capstor/scratch/cscs/omlins/todi/juliaup/depot
[ JUHPC: JULIA_DEPOT_PATH: /capstor/scratch/cscs/omlins/todi/juliaup/depot
[ JUHPC: JULIA_LOAD_PATH: :/capstor/scratch/cscs/omlins/todi/juhpc_setup/julia_preferences
[ JUHPC: CUDA_HOME: /opt/nvidia/hpc_sdk/Linux_aarch64/23.9/cuda/12.2
[ JUHPC: JULIA_CUDA_MEMORY_POOL: none
[todi][omlins@todi-ln002 ~]$ julia --version
julia version 1.10.5
[todi][omlins@todi-ln002 ~]$
I cannot make any sense of your latest output. Can you try again with a fresh login using the latest version on main?
I did that. (Deleted all existing directories, downloaded juhpc afresh, new shell.) This is the output:
$ bash -l /tmp/juhpc /Users/eschnett/juhpc_setup /Users/eschnett/juliaup
/Users/eschnett/.bash_profile: line 84: /Users/eschnett/miniforge3/etc/profile.d/conda.sh: No such file or directory
██╗ ██╗ ██╗ ██╗ ██╗██████╗ ██████╗
██║ ██║ ██║ ██║ ██║██╔══██╗██╔════╝
██║ ██║ ██║ ███████║██████╔╝██║
██╗ ██║ ██║ ██║ ██╔══██║██╔═══╝ ██║
╚█████╔╝ ╚██████╔╝ ██║ ██║██║ ╚██████╗
╚════╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═════╝
[ JUHPC: Starting installation of HPC setup for juliaup, julia and HPC key packages requiring system libraries...
[ JUHPC: Installing temporary juliaup installation in /dev/shm/eschnett/juliaup...
info: downloading installer
Welcome to Julia!
This will download and install the official Julia Language distribution
and its version manager Juliaup.
Juliaup will be installed into the Juliaup home directory, located at:
/dev/shm/eschnett/juliaup
The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:
/dev/shm/eschnett/juliaup/bin
You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.
Now installing Juliaup
Error: Failed to create install folder for Juliaup.
Caused by:
Operation not permitted (os error 1)
[ JUHPC: ERROR: failed to install Juliaup (and Julia) in /dev/shm/eschnett/juliaup.
[ JUHPC: cleaning up temporary juliaup installation in /dev/shm/eschnett/juliaup.
I notice that I specify my home directory /Users/eschnett/juliaup, and this then becomes /dev/shm/eschnett/juliaup. That's a bit weird.
Also, I'm on macOS, with
$ ls /dev/shm
ls: /dev/shm: No such file or directory
I didn't realize that before.
When I replace /dev/shm by /tmp, things seem to work. juliaup (i.e. Julia 1.10) is installed into /tmp/eschnett/juliaup.
Weirdly, when I then activate juhpc and run julia, Julia 1.10 is installed again! That's unexpected.
ls: /dev/shm: No such file or directory
I added a check to use /tmp when this is the case
I notice that I specify my home directory /Users/eschnett/juliaup, and this then becomes /dev/shm/eschnett/juliaup. That's a bit weird.
No that's only the temporary installation which is used to create the setup that goes there.
Weirdly, when I then activate juhpc and run julia, Julia 1.10 is installed again! That's unexpected.
When the end user calls juliaup for the first time, it is installed in the directory chosen (In your case /Users/eschnett/juliaup). The end user does not have to be the same as the user or admin that runs JUHPC. This means for you concretely, for example, that you can create a Julia HPC setup in a folder where all the members of your project have access to. Then the other members of your project only need to activate the script from the folder to use the setup that you have created (they don't need to run JUHPC). Nevertheless, each member will have his own installation, as long as you use $USER instead of eschnett etc. in the JUHPC_INSTALLDIR argument to juhpc.
@eschnett Based on your feedback I further improved the output and some added some sanity checks. Does it now work all fine for you as well? Then, I will do a release.
New release done