Installation bug (?) in Colab
jqbond opened this issue · 16 comments
First, I want to thank you for developing this. I use it for teaching as I don't have the ability to add a local Jupyter/Julia installation to the various classroom computers. I used it extensively about a year ago by installing Julia 1.7.2, IIRC.
I am teaching this course again, and I went to run the Julia installation, and it is throwing an error that I can't quite wrap my head around. I am guessing it is on the Colab side. Any guidance on a resolution would be appreciated!!
Installing Julia 1.8.2 on the current Colab Runtime...
2023-01-20 15:45:13 URL:https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz [135859273/135859273] -> "/tmp/julia.tar.gz" [1]
Installing Julia package IJulia...
/bin/bash: line 24: 229 Aborted (core dumped) julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
[<ipython-input-1-9b4dbd3cbe07>](https://localhost:8080/#) in <module>
----> 1 get_ipython().run_cell_magic('shell', '', 'set -e\n\n#---------------------------------------------------#\nJULIA_VERSION="1.8.2" # any version ≥ 0.7.0\nJULIA_PACKAGES="IJulia BenchmarkTools"\nJULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions\nJULIA_NUM_THREADS=2\n#---------------------------------------------------#\n\nif [ -z `which julia` ]; then\n # Install Julia\n JULIA_VER=`cut -d \'.\' -f -2 <<< "$JULIA_VERSION"`\n echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."\n BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"\n URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"\n wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"\n tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1\n rm /tmp/julia.tar.gz\n\n # Install Packages\n nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0\n if [ $GPU -eq 1 ]; then\n JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"\n fi\n for PKG in `echo $JULIA_PACKAGES`; do\n echo "Installing Julia package $PKG..."\n julia -e \'using Pkg; pkg"add \'$PKG\'; precompile;"\' &> /dev/null\n done\n\n # Install kernel and rename it to "julia"\n echo "Installing IJulia kernel..."\n julia -e \'using IJulia; IJulia.installkernel("julia", env=Dict(\n "JULIA_NUM_THREADS"=>"\'"$JULIA_NUM_THREADS"\'"))\'...
2 frames
[/usr/local/lib/python3.8/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in check_returncode(self)
132 def check_returncode(self):
133 if self.returncode:
--> 134 raise subprocess.CalledProcessError(
135 returncode=self.returncode, cmd=self.args, output=self.output)
136
CalledProcessError: Command 'set -e
#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#
if [ -z `which julia` ]; then
# Install Julia
JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
rm /tmp/julia.tar.gz
# Install Packages
nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
if [ $GPU -eq 1 ]; then
JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
fi
for PKG in `echo $JULIA_PACKAGES`; do
echo "Installing Julia package $PKG..."
julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
done
# Install kernel and rename it to "julia"
echo "Installing IJulia kernel..."
julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
"JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
mv -f $KERNEL_NAME "$KERNEL_DIR"/julia
echo ''
echo "Successfully installed `julia -v`!"
echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
echo "jump to the 'Checking the Installation' section."
fi
' returned non-zero exit status 134.
@ageron I am seeing this too, across all Julia versions
Thanks for your feedback, I'll look into this issue right now.
Yikes, it's indeed completely broken. It looks like it's not due to the installation code, but rather some change in Colab's runtime environment which prevents Julia from running. See this discussion. It's not a simple fix on my end, unfortunately. I'll keep looking, but it may require some patch from Google, and since they don't officially support Julia, they might have very little bandwidth for this. I hope it's not the end of Julia on Colab, it's really convenient.
@ageron that was my guess when I first encountered the error. I ended up borrowing some of the Jupyter Docker Stacks and then customizing a Dockerfile with Julia packages. I'm running it now in Codespaces on GitHub as my cloud-based Julia-in-a-Notebook solution. Let me know if that is of interest, and I can link you to the repository.
Hey @ageron @jqbond there are some helpful solutions being posted here: googlecolab/colabtools#3385. It sounds like they are trying to patch this in the next runtime update which is good.
It's great that the Colab team has acknowledged this bug and will try to fix it. I've tried to run "Use fallback runtime version" from the Command palette, as they suggested, but unfortunately I got a message saying that the fallback runtime is unavailable.
However, the second solution they suggested worked: I just replaced every julia command in the first cell of the notebooks with LD_PRELOAD= julia (note the space before julia), and then things ran normally.
I'll commit this change and revert it once they've fixed Colab.
@ageron Thanks for the solution -- I'll make a note of it. I also created a bare bones repository that should run Julia/Jupyter in a Codespace. The default configuration should open up a VS Code environment, and you can run Jupyter tools from the terminal, e.g., jupyter lab
Link below.
https://github.com/jqbond/Julia_Codespace
**Edit: I've been running with an 8-Core virtual machine, which you can change when you configure the codespace. Can't guarantee adequate memory with 2 or 4 core, but you can try them out.
Hey @ageron @jqbond everything is fixed with the current runtime now – no need for the fallback runtime or LD_PRELOAD=. See googlecolab/colabtools#3385.
That was much faster than I expected, cool! I removed LD_PRELOAD= , everything seems to work fine now. Thanks for your feedback everyone. 👍
Has this error shown up again? Because I am getting this output:
CalledProcessError: Command 'set -e ' returned non-zero exit status 8.
Yesterday everything was fine.
I just looked and the error is this:
tar: This does not look like a tar archive
so not a colab-specific error? I think the default URLs need to be updated. Perhaps juliaup could be a good choice, since they have a non-interactive installation mode now.
My code cell is this
%%shell
set -e
#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#
if [ -z `which julia` ]; then
# Install Julia
JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
rm /tmp/julia.tar.gz
# Install Packages
nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
if [ $GPU -eq 1 ]; then
JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
fi
for PKG in `echo $JULIA_PACKAGES`; do
echo "Installing Julia package $PKG..."
julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
done
# Install kernel and rename it to "julia"
echo "Installing IJulia kernel..."
julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
"JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
mv -f $KERNEL_NAME "$KERNEL_DIR"/julia
echo ''
echo "Successfully installed `julia -v`!"
echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
echo "jump to the 'Checking the Installation' section."
fi
and the output is this:
Installing Julia 1.8.2 on the current Colab Runtime...
https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz:
2023-04-08 23:13:56 ERROR 403: Forbidden.
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
[<ipython-input-6-5e5c48dc9892>](https://localhost:8080/#) in <cell line: 1>()
----> 1 get_ipython().run_cell_magic('shell', '', 'set -e\n\n#---------------------------------------------------#\nJULIA_VERSION="1.8.2" # any version ≥ 0.7.0\nJULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"\nJULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions\nJULIA_NUM_THREADS=2\n#---------------------------------------------------#\n\nif [ -z `which julia` ]; then\n # Install Julia\n JULIA_VER=`cut -d \'.\' -f -2 <<< "$JULIA_VERSION"`\n echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."\n BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"\n URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"\n wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"\n tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1\n rm /tmp/julia.tar.gz\n\n # Install Packages\n nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0\n if [ $GPU -eq 1 ]; then\n JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"\n fi\n for PKG in `echo $JULIA_PACKAGES`; do\n echo "Installing Julia package $PKG..."\n julia -e \'using Pkg; pkg"add \'$PKG\'; precompile;"\' &> /dev/null\n done\n\n # Install kernel and rename it to "julia"\n echo "Installing IJulia kernel..."\n julia -e \'using IJulia; IJulia.installkernel("julia", env=Dict(\n ...
3 frames
[/usr/local/lib/python3.9/dist-packages/google/colab/_shell.py](https://localhost:8080/#) in run_cell_magic(self, magic_name, line, cell)
331 if line and not cell:
332 cell = ' '
--> 333 return super().run_cell_magic(magic_name, line, cell)
334
335
[/usr/local/lib/python3.9/dist-packages/IPython/core/interactiveshell.py](https://localhost:8080/#) in run_cell_magic(self, magic_name, line, cell)
2471 with self.builtin_trap:
2472 args = (magic_arg_s, cell)
-> 2473 result = fn(*args, **kwargs)
2474 return result
2475
[/usr/local/lib/python3.9/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in _shell_cell_magic(args, cmd)
110 result = _run_command(cmd, clear_streamed_output=False)
111 if not parsed_args.ignore_errors:
--> 112 result.check_returncode()
113 return result
114
[/usr/local/lib/python3.9/dist-packages/google/colab/_system_commands.py](https://localhost:8080/#) in check_returncode(self)
135 def check_returncode(self):
136 if self.returncode:
--> 137 raise subprocess.CalledProcessError(
138 returncode=self.returncode, cmd=self.args, output=self.output
139 )
CalledProcessError: Command 'set -e
#---------------------------------------------------#
JULIA_VERSION="1.8.2" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools JUDI PyPlot ColorSchemes Statistics ImageFiltering"
JULIA_PACKAGES_IF_GPU="CUDA" # or CuArrays for older Julia versions
JULIA_NUM_THREADS=2
#---------------------------------------------------#
if [ -z `which julia` ]; then
# Install Julia
JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
rm /tmp/julia.tar.gz
# Install Packages
nvidia-smi -L &> /dev/null && export GPU=1 || export GPU=0
if [ $GPU -eq 1 ]; then
JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
fi
for PKG in `echo $JULIA_PACKAGES`; do
echo "Installing Julia package $PKG..."
julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
done
# Install kernel and rename it to "julia"
echo "Installing IJulia kernel..."
julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
"JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
mv -f $KERNEL_NAME "$KERNEL_DIR"/julia
echo ''
echo "Successfully installed `julia -v`!"
echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
echo "jump to the 'Checking the Installation' section."
fi
' returned non-zero exit status 8.
Yeah, the URL doesn't exist anymore: go to https://storage.googleapis.com/julialang2/bin/linux/x64/1.8/julia-1.8.2-linux-x86_64.tar.gz (i.e., it's not a colab issue). Maybe raise a new issue instead?
The default URL needs to be updated. Or just changed to juliaup instead.
Ok, now, how do I install Julia with juliaup?