Feature: Using SDKMAN in a CI pipeline

[lbruun]

Clearly, SDKMAN was (originally) designed for interactive and personal use by a developer.
It can be used in a CI pipeline, but such use case under-documented. This is sad as SDKMAN is a great tool.

This ticket explores how SDKMAN can be used in such context and also what might be improved in SDKMAN (in code or documentation) to make such adoption easier.

CI docker image with SDKMAN

Many CI systems use Docker for execution and therefore require that you put your tools (e.g. SDKMAN) into a docker image. SDKMAN is both useful at build time (for installing software that we want included the image) as well as at execution time (to allow a CI pipeline to install additional software).

Putting SDKMAN into a Docker image for such purpose is in fact not that simple as there as many things to remember. Below is an example of how it might be done:

FROM ubuntu:22.04

# (1)
SHELL ["/bin/bash", "-c"]

# (2) 
# Install needed packages
RUN apt-get update && apt-get install -y \
  curl \
  unzip \
&& rm -rf /var/lib/apt/lists/*

# (3)
# Install SDKMAN  
RUN curl -s -S -o ./get-sdkman.sh "https://get.sdkman.io?rcupdate=false" \
  && chmod +x ./get-sdkman.sh \
  && ./get-sdkman.sh \
  && rm ./get-sdkman.sh \
  # Change SDKMAN's config to be suitable for a CI pipeline execution
  && sed -i 's/sdkman_auto_answer=false/sdkman_auto_answer=true/g'                $HOME/.sdkman/etc/config \
  && sed -i 's/sdkman_selfupdate_feature=true/sdkman_selfupdate_feature=false/g'  $HOME/.sdkman/etc/config \
  && sed -i 's/sdkman_colour_enable=true/sdkman_colour_enable=false/g'            $HOME/.sdkman/etc/config 

# (4)
ENV BASH_ENV="\$HOME/.sdkman/bin/sdkman-init.sh"

# (5)
ENTRYPOINT ["/bin/bash", "-c"]

Comments to Dockerfile:

1. SDKMAN requires Bash. The SHELL statement governs the shell used for subsequent RUN commands in the Dockerfile, not the shell used in the finished image.
2. I'm a bit uncertain about what packages SDKMAN requires. It is not documented, AFAIK. This is my guess.
3. Installing SDKMAN:
   • We use rcupdate=false as having SDKMAN manipulate .bashrc file is pointless for our use case. It wouldn't hurt, it would just be pointless.
   • There is no CI friendly install of SDKMAN so we have to manipulate the SDKMAN config file post installation.
4. SDKMAN needs to be in path and so on. This is taken care of by the sdkman-init-sh script. So far, so good. You can get SDKMAN to add this to .bashrc file but that file is only sourced for interactive logins. So for non-interactive logins you have to take care of this yourself. Using the BASH_ENV variable is one way to solve this problem. Notice the escaped $ character. This is important as we want the HOME variable to be resolved when the image is executed, not when it is built.
5. Make sure Bash is used.

You can of course use SDKMAN to install software in the docker image. The Dockerfile snippet can for example look like this (example installs Java 21):

RUN sdk install java 21.0.1-tem  \
  && sdk flush 

Notice how we execute sdk flush after the installation. This is in order to keep the size of the image as low as possible. It is crucial that the BASH_ENV variable is defined in the Dockerfile prior to attempting to use the sdk command in RUN statements. The Dockerfile ENV statement has effect both at build time and run time.

What might be improved in SDKMAN?

Here are some suggestions:

Suggestion 1

Have an install feature which creates a CI friendly config file, e.g. with

sdkman_auto_answer=true
sdkman_selfupdate_feature=false
sdkman_colour_enable=false

Suggestion 2

Document what packages (e.g. curl, etc) that SDKMAN requires for its execution.

Suggestion 3

Document how SDKMAN might be used in a Dockerfile. Have some examples in the documentation. This ticket is hopefully a good starting point.

[alkaphreak]

Suggestions :

• add cmd like git config to set sdkman config option
• add a -q (quiet) or -y (yes) -n (no) command line option to go in non interactive mode
• add one liner options to combine commands like sdk install java lts -q --flush

[marc0der]

Hi @lbruun, I'm so sorry that I never saw your issue before and never replied. I only became aware of it when @alkaphreak posted.

So if you still want some of this done, we can look into it. Here is my response to each of your points:

1. We can populate a CI-friendly config file. This can be done by adding a request parameter like ?ci=true onto the URL on installation. We could spin up a separate ticket to describe what values should be in the config file.
2. Docs are easy to update, so we can certainly look into stating those dependencies explicitly on the installation page of our site.
3. So here we could go in two directions. We could write some docs to describe how to run SDKMAN in a Dockerfile (perhaps you could even contribute this in our wiki). Alternatively, we could roll a ready-made Docker image that runs off some recipe (thinking a sdkmanrc file that specifies what to install into the container when it is run, and then that could be added as a layer to bake a new image. I'm just thinking off the top of my head here.

I'd be interested to hear your thoughts on this.

@alkaphreak regarding your points:

1. We already have this. It's called sdk config.
2. Also already supported by the sdkman_auto_answer=true|false config as documented on our website.
3. The flush is actually not needed, as the only residual files left behind in the tmp folder are scripts used for installation (a few k of space). No binaries are left behind, they are automatically removed on extraction on your system. I feel it would pollute the sdk install command if we had to add an unrelated --flush option, which has nothing to do with installation. Lastly, the --quiet is something we will most certainly look at during the Rust rewrite of install, as well as the tracking of latest and lts versions of Java.

[alkaphreak]

@marc0der the git config option=value give the possibility to set a config value without opening the config file. It's then easier to use within a script or a CI.

[lbruun]

@marc0der:

We can populate a CI-friendly config file. This can be done by adding a request parameter like ?ci=true onto the URL on installation. We could spin up a separate ticket to describe what values should be in the config file.

Yes, please do that. I would call it ?unattended=true (or similar) because that is what it would do: it configures SDKMAN to be used in an non-interactive environment.

So here we could go in two directions. We could write some docs to describe how to run SDKMAN in a Dockerfile (perhaps you could even contribute this in our wiki). Alternatively, we could roll a ready-made Docker image that runs off some recipe (thinking a sdkmanrc file that specifies what to install into the container when it is run, and then that could be added as a layer to bake a new image. I'm just thinking off the top of my head here.

There are two scenarios:

• #1 People who are happy to download the SDKMAN as part of their CI pipelines. I've always found this a bit lazy because you effectively pay the price of download/unpack/install for each and every pipeline execution. But you see it a lot in CI pipelines. Even though this is not my taste, SDKMAN should make this scenario easy. (I guess there a CI platforms that will not let you use an alternative docker image for the job execution and this may explain why CI engineers are sometimes forced into this solution).

• #2 The other scenario is that SDKMAN is to be baked into some docker image. SDKMAN would rarely be the only tool in such image. Personally, when I need to build a multi-tool docker image I want to start out from an image that I know will not suddenly morph into something unexpected in the future. So I build from some OS base, like FROM ubuntu rather than picking (at random) the base image produced by one of the tools I want included. As an example, let's say I wanted to build an image with both SDKMAN and kubectl and awscli in one image. In such case, even if the SDKMAN project had published a docker image, I would not use it as my base. Instead, I would start from an "neutral base". But that's just me. So this is not to say that a docker image published by the SDKMAN project would be completely useless. SDKMAN can fetch a lot of tools on-the-fly so for those who are happy with the download/unpack/install cycle for every job invocation it would make sense.

[marc0der]

Hey @lbruun,

I would call it ?unattended=true (or similar) because that is what it would do: it configures SDKMAN to be used in a non-interactive environment.

The current installation is also unattended but creates a config for a developer machine. For that reason, I prefer ?ci=true because it describes the actual use case in a CI environment. It's a minor implementation detail, which should still work for you. Who said naming was easy? 😄

As for 3, point taken. I'll document the new flag clearly once I've implemented it. We can always revisit a custom Docker image if someone has the requirement, but we'll leave it be for now.