eskimo: A Java repository from Eskimo.sh

1. Eskimo Introduction

A state of the art Big Data Infrastructure and Management Platform to build, manage and operate Big Data 2.0 Analytics clusters on Kubernetes.

Eskimo is in a certain way the Operating System of your Big Data Cluster:

A plug and play, working out of the Box, Big Data Analytics platform fulfilling enterprise environment requirements.
A state of the art Big Data 2.0 platform
- based on Kubernetes (and heavily relying on Docker and SystemD as well)
- packaging Gluster, Spark, Kafka, Flink, ElasticSearch
- with all the administration and management consoles such as Cerebro, Kibana, Zeppelin, Kafka-Manager, Grafana, Prometheus and of course the Kubernetes Dashboard.
- everything all 100% pre-configured and setup to work perfectly together.
An Administration Application aimed at drastically simplifying the deployment, administration and operation of your Big Data Cluster
A Data Science Laboratory and Production environment where Data Analytics is both
- developed and
- operated in production

Eskimo is as well:

a collection of ready to use docker containers packaging fine-tuned and highly customized plug and play services with all the nuts and bolts required to make them work well together.
a framework for developing, building and deploying Big Data and NoSQL services based on Kubernetes, Docker and SystemD.

Eskimo Packages all the components above and takes care of all the configuration and tuning required to make them work together smoothly and without the user or administrator needing to take care of anything, Big Data the Plug & Play way.

1.1. Eskimo CE Project Development Status

GitHub Action CI Integration Build Status

SonarCloud Quality Checks Status

2. Information

Reach http://www.eskimo.sh for more information on Eskimo or look at the documentation in the folder doc.

3. Building eskimo

3.1. Requirements

Eskimo uses plain old Apache Maven for building (https://maven.apache.org/) as well as JDK 11+ (https://openjdk.java.net/) for both building and execution.

Minimum requirements are as follows:

JDK 11 or greater
Apache maven 3.5 or greater.

Every other dependency will be downloaded during the maven build process (except trilead-ssh2, see Install maven dependency trilead-ssh2 below).

Theoretically, Eskimo can be built on any Operating System providing a command line and supporting Java 11+ (Note : previous versions of JDK down to 8 may be supported without any guarantee) and Maven 3+.

3.1.1. Microsoft Windows

However, the eskimo source tree makes an extensive use of symbolic links which - unfortunately - are only supported on Microsoft Windows 10 and 11 through "Developer Mode".

In order to checkout and develop Eskimo on Windows, some precautions need to be taken:

First, Developer Mode should be enabled on Windows to support symbolic links.
See https://github.com/MicrosoftDocs/windows-dev-docs/blob/docs/hub/apps/get-started/developer-mode-features-and-debugging.md
Second, The git clone command should explicitly enable symbolic link support. That would mean using following command for instance to checkout Eskimo master branch:
git clone -c core.symlinks=true https://github.com/eskimo-sh/eskimo.git

If these two precautions are well taken, one should be able to develop Eskimo on Windows.
A significant portion of the unit tests will however be skipped when executed on Windows since they require a Unix Operating System to work properly.

3.2. Building using maven

In order to build eskimo, simply run the following command in this very folder (the one containing this readme.adoc file):

Build Eskimo

mvn clean install

As a result of the build process, Eskimo is available as an extractable archive in eskimo-version-bin.zip or eskimo-version-bin.tar.gz in the folder target.

3.3. Specific hints and troubleshooting information regarding eskimo building

You might want to read carefully the following additional information related to building eskimo.

3.3.1. Put maven and Java in PATH

Of course, for the above command to work, you need to have java and mvn in your path.

Use for instance the following commands on Linux:

Put maven and java in PATH on Linux

export JAVA_HOME=/usr/local/lib/jdk-11  ## (or wherever it's installed)
export MAVEN_HOME=/usr/local/lib/apache-maven-3.5.3  ## (same)
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

(You might want to put above commands in your /etc/profile or /etc/bash.bashrc)

And for instance the following commands on Windows

Put maven and java in PATH on Windows

set JAVA_HOME=C:\programs\jdk-11  :: (or wherever it's installed)
set MAVEN_HOME=C:\programs\apache-maven-3.6.1  :: (same)
set PATH=%MAVEN_HOME%\bin;%JAVA_HOME%\bin;%PATH%

3.3.2. Install maven dependency trilead-ssh2

Eskimo used the library trilead-ssh2 version build-217-jenkins-27 which is not available in standard maven repositories.
This library is bundled with the eskimo sources in the folder lib.
One can install it in his own local maven repository using the following command:

Install trilead-ssh2 library

cd libs
bash install_libs.sh

3.4. Eskimo Build result

The build results at the end of the maven build process is located in the folder `target.
The results are two archive files:

A zip archive : eskimo-version-bin.zip
A tarball archive : eskimo-version-bin.tar.gz

You can find pre-built packages of Eskimo on https://www.eskimo.sh.

The archive contains the full software packages with all dependencies and command line executables required to start it as well as the full documentation, etc.

4. Running Eskimo

This section gives instructions about running eskimo.

Eskimo itself can run on any Operating System supporting a JVM (Linux, Windows, Mac OSX) but cluster nodes are only supported running Linux.

4.1. Prerequisites

4.1.1. System requirements

In order to run eskimo, one needs to have

At least 20Gb of disk storage space on the machine running Eskimo
At least one linux machine available on the network (can be the same machine than the one running Eskimo) with either a debian-based (Ubuntu, Debian, etc.), red-hat-based (CentOS, RHEL, etc.) or OpenSUSE Operating System.

(See The Eskimo User Guide for more information on Eskimo cluster requirements.)

Eskimo is reached using a web browser (see startup logs). Supported web browsers are:

Microsoft Edge 14 or greater
Mozilla FireFox 54 or greater
Google Chrome 58 or greater

Note: there may be other browsers / versions supported (Safari, Opera but they are not certified to work with Eskimo)

4.1.2. Available JVM with Java in PATH

In order to run eskimo, one needs to have java in the path.

Use for instance the following commands on Linux:

Put java in PATH on Linux

export JAVA_HOME=/usr/local/lib/jdk-11
export PATH=$JAVA_HOME/bin:$PATH

(You might want to put above commands in your /etc/profile or /etc/bash.bashrc)

And for instance the following commands on Windows

Put java in PATH on Windows

set JAVA_HOME=C:\programs\jdk-11
set PATH=%JAVA_HOME%\bin;%PATH%

4.2. Extract archive.

After building eskimo using maven, the zip and tarball archives are located in the folder target.

One of these archives needs to be extracted on the local filesystem.

Then in the folder bin under the newly extracted eskimo binary distribution folder, one can find two scripts:

a script eskimo.bat to execute eskimo on Windows
a script eskimo.sh to execute eskimo on Linux.

4.3. Access eskimo

With eskimo properly started using the above scripts, one can reach eskimo using http://machine_ip:9191.
The default port number is 9191. This can be changed in configuration file eskimo.properties.

The default login / password credentials are admin / password.

4.4. Typical startup issues

Several issues can happen upon first eskimo startup.
This section describes common issues and ways to resolved them.

4.4.1. eskimo-users.json cannot be written

If you meet an error as the following on startup:

Impossible to write eskimo-users.json

Caused by: ch.niceideas.common.utils.FileException: ./eskimo-users.json (Unauthorized access)
        at ch.niceideas.common.utils.FileUtils.writeFile(FileUtils.java:154)
        at ch.niceideas.eskimo.security.JSONBackedUserDetailsManager.<init>(JSONBackedUserDetailsManager.java:81)
        at ch.niceideas.eskimo.configurations.WebSecurityConfiguration.userDetailsService(WebSecurityConfiguration.java:127)
        ... 50 more
Caused by: java.io.FileNotFoundException: ./eskimo-users.json (Unauthorized access)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:276)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:220)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:170)
        at java.base/java.io.FileWriter.<init>(FileWriter.java:90)
        at ch.niceideas.common.utils.FileUtils.writeFile(FileUtils.java:149)
        ... 52 more

Eskimo uses a local file to define users and access credentials. Upon first startup, if that file doesn’t exist already, it is created by eskimo (with the default credentials above) at the path pointed to by the property security.userJsonFile in eskimo.properties.

If you experience the error above or something alike, change that propery to point to a location where the first version of the file can successfully be created.

4.5. Building packages / docker images locally

With released versions of Eskimo, operators / administrators have the choice to either build packages / docker images locally - which boils down to creating container images where the target software (e.g. spark, flink, gluster, etc.) is downloaded from internet and installed - or download pre-built images from www.eskimo.sh.
Whereas on development - SNAPSHOT - versions of Eskimo downloading pre-built images is not possible, only the possibility to build images locally is available.

Whenever building images locally, the majority of the failures from from references versions of software components not avaiilable anymore. For instance, when the Apache flink team releases a new minor version, they most of the time remove the previous minor from the download site.
When such think happen the building of the corresponding local image will fail in Eskimo.
The only solution to this is to update the target version in the file packages_dev/common/common.sh.

5. Eskimo source distribution layout

The directory structure of the eskimo source distribution is as follows:

doc contains the source documentation in asciidoc format
libs contains dependencies not available in standard maven repositories
packages_dev contains the docker images development framework and packages
packages_distrib is the destination folder in which downloaded or built images are placed
services_setup contains the services installation framework and packages
src contains the source files to build eskimo
test_lab contains various tools to build VMs aimed at testing eskimo

6. Test Laboratory

The folder test_lab folder in the Eskimo CE project root folder contains a Vagrant framework used to create virtual machines to test Eskimo.

The supported virtual machines are declared in the file VagrantFile in the variable nodes:

Vagrant Virtual Machines declaration

# Define cluster nodes
nodes = [
  { :hostname => 'deb-node1',  :box => 'debian/buster64',
                               :ip => '192.168.10.11', :ram => 8000 },
  { :hostname => 'deb-node2',  :box => 'ubuntu/xenial64',
                               :ip => '192.168.10.12', :ram => 6000 },
  { :hostname => 'cent-node1', :box => 'centos/7',
                               :ip => '192.168.10.13', :ram => 6000},
  { :hostname => 'cent-node2', :box => 'fedora/29-cloud-base',
                               :ip => '192.168.10.14', :ram => 6000 }
]

The defined VMs use different Operating Systems to test Eskimo’s installation on different OSes.
One can edit this variable to declare one’s own VM’s as required.

An individual VM is then started with vargrant using VirtualBox as follows:

Start deb-node1 with VirtualBox

$ cd ./test_lab/vagrant/
$ vagrant up deb-node1

Vagrant can also start VM’s using libvirt / QEMU / kvm if the required libraries and provider are properly installed:

Start deb-node1 with libvirt

$ cd ./test_lab/vagrant/
$ vagrant up deb-node1 --provider=libvirt

6.1. Integration tests

A script integration-test.sh is provided in test_lab/integration_test that is instrumental in building and testing eskimo in depth.
It enables to test rebuilding a full fledged Eskimo environment from scratch on a test platform of one single of multiple node(s) (VMs) and test all its features (run all the zeppelin sample notebooks, ensure proper deployment of all services, etc.).
A specific command line flag in this script even enables to build a full Demo VM from scratch (See user guide on Demo VM).

It’s usage is at follows:

integration-test.sh

eskimo@notebook:/work/eskimo/test_lab/integration_test$ ./integration-test.sh -h
Usage: integration-test.sh [Options] [Target Box IP]
  where [Target Box IP] is optional IP address of box to target tests at
  where [Options] in
    -h  Display this help message.
    -p  Rebuild eskimo service packages
        -n  Skip packages rebuild (useful when used with -a)
    -r  Rebuild eskimo (otherwise take latest build)
        -f  Fast repackage
    -b  Recreate box(es)
    -e  (Re-)install Eskimo on box
    -s  Setup Eskimo
    -l  Run Data load
    -z  Run Zeppelin notebooks test
    -i  Run CLI tests
    -t  Run other tests
    -c  Run cleanup
    -o  Take screenshots
    -a  RUN ALL OF THE ABOVE
    -w  Use screenshots to overwrite git tree images
    -d  Prepare the VM for DemoVM
    -m  Test on multiple nodes

For instance, the DemoVM is built using the following call:

DemoVM building with integration-test.sh

eskimo@notebook:/work/eskimo/test_lab/integration_test$ ./integration-test.sh -ad

Multiple dependencies are required on the machine where this `integration-test.sh `script is intended to run:

Vagrant 2.2+
VirtualBox 6+
Apache Maven 3+
Java 11+
Docker 20+
OpenSSH and sshpass

All of these needs to be available in the PATH. (Lower versions of these software components might work as well but without any guarantee).

7. Further information

Look at the following files for more information

The Eskimo User Guide
readme.adoc in the folder packages_dev for information about the components docker images building framework
readme.adoc in the folder services_setup for information about the services installation and operation framework.

Appendix A: Copyright and License

Eskimo is available under a dual licensing model : commercial and GNU AGPL.
If you did not acquire a commercial licence for Eskimo, you can still use it and consider it free software under the terms of the GNU Affero Public License. You can redistribute it and/or modify it under the terms of the GNU Affero Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Compliance to each and every aspect of the GNU Affero Public License is mandatory for users who did no acquire a commercial license.

Eskimo is distributed as a free software under GNU AGPL in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero Public License for more details.

You should have received a copy of the GNU Affero Public License along with Eskimo. If not, see https://www.gnu.org/licenses/ or write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA, 02110-1301 USA.

You can be released from the requirements of the license by purchasing a commercial license. Buying such a commercial license is mandatory as soon as :

you develop activities involving Eskimo without disclosing the source code of your own product, software, platform, use cases or scripts.
you deploy eskimo as part of a commercial product, platform or software.

For more information, please contact eskimo.sh at https://www.eskimo.sh

The above copyright notice and this licensing notice shall be included in all copies or substantial portions of the Software.

eskimo-sh/eskimo