Danube Cloud :: Factory

This repository is used for building various components of Erigones Danube Cloud, as well as the final Danube Cloud USB key image.

Output

All build results are stored in following folders in build_base_dir:

appliances: Danube Cloud appliances (VM images), which then become part of the Danube Cloud USB image
- base-64-es
- esdc-mon
- esdc-mgmt
- esdc-cfgdb
- esdc-dns
- esdc-img
isos: ISO images, which are downloaded from external sources and become part of the Danube Cloud USB image.
archives: Tarballs, which become part of the Danube Cloud USB image (/opt)
- local
- monitoring
- opt-custom
- esdc-node
platform: Tarballs, which contain the hypervisor platform - SmartOS raw kernel and boot_archive.
images: This is just a directory, which holds symlinks to all appliances in an IMGAPI-like folder structure.
usb: Danube Cloud USB images.

Requirements

Two network-connected Linux/Unix machines are required:

buildnode remote host:
- A SmartOS/ErigonOS machine that is capable of running KVM and OS virtual machines.
- Required software:
  
  Python 2.7+
builder local host:
- The machine (can be a VM), which has this repository checked out.
- Required software:
  
  git
  
  Ansible >= 2.3
  
  GNU make (gmake)
  
  sshpass
  
  OpenSSH client
  
  a working ssh-agent with loaded build_ssh_key (for running git clone on remote host)
  
  a running web server (build_base_url)
- Must have a web root directory ready (build_base_dir) and the directory structure must be browsable (autoindex in nginx), e.g.:
  server { listen 80; server_name _; root /var/www/html; # build_base_url autoindex on; location /images { index manifest; default_type application/json; } location ~ /images/(.*)/(file|.*.zfs.gz)$ { default_type application/octet-stream; } }
- NOTE: Building the usb-image target currently requires an ErigonOS/SmartOS machine or a zone with fs_allowed property set to "ufs,pcfs,tmpfs".

Create build environment

Create builder VM (on plain SmartOS or on Danube Cloud):

It is recommended to use the same image that is used for building official SmartOS: base-64-lts 18.4.0 with UUID c193a558-1d63-11e9-97cf-97bb3ee5c14f. But you can use any SunOS image, just follow the requirements for builder from previous sections. This sections will describe the recommended setup.
8 GB RAM, at least 20GB of disk space. Use of ZLE disk compression is recommended.
Add a delegated dataset (optional, for builds speed up)

Modify fs_allowed` property from the global zone:

vmadm update <vm_uuid> fs_allowed="ufs,pcfs,tmpfs"
vmadm reboot <vm_uuid>

Log into the VM and set up packages (as root user):

pkgin up
pkgin fug
pkgin in git gmake ansible sshpass nginx
ssh-keygen -t ecdsa
mkdir /data
cd /data        # this is build_base_dir
git clone https://github.com/erigones/esdc-factory.git
cd esdc-factory/etc
cp config.sample.yml config.yml
cp hosts.sample.cfg hosts.cfg
cd ..

edit etc/config.yml:
- build_base_url - use the IP address of this builder VM (e.g. http://10.111.10.206)
- build_base_dir - /data by default
- build_ssh_key - content of ~/.ssh/id_ecdsa.pub on the builder VM. This ssh key needs to be pushed to buildnode (SmartOS global zone).
- build_ip - IP address of a temporary VM that will be created during image builds
- build_gateway, build_netmask, build_nic_tag, build_vlan_id - network settings that will be used by temporary VMs during image builds
edit etc/hosts.yml:
- edit IP address of buildnode (SmartOS global zone that will be used for creating VMs)
push ssh public key from ~/.ssh/id_ecdsa.pub to the buildnode's /root/.ssh/authorized_keys so the buildnode VM can access the builder without a password

configure and enable nginx on the buildnode VM. You can find the sample nginx config in etc/nginx.conf.sample:

cp -f /data/esdc-factory/etc/nginx.conf.sample /opt/local/etc/nginx/nginx.conf
svcadm enable nginx
svcs nginx

initialize factory on builder VM

cd /data/esdc-factory
eval `ssh-agent -s`
ssh-add
gmake init

Usage

$ make help

Examples (more examples are in make help):

$ eval `ssh-agent -s`
$ ssh-add
$ make base-64-es
$ make base-centos-7
$ make archives
$ make isos
$ make platform     # this needs setup of smartos compile environment
$ make esdc
$ env EXTRA_VARS="usb_type=cn" gmake usb-image
$ env EXTRA_VARS="usb_type=hn" gmake usb-image

** See also: docs/make-help.txt

Parallel builds

If you want to use parallel builds, you need to specify multiple temporary IPs and multiple VNC ports so the VMs won't create collisions. Define build_ips and build_vnc_ports dicts with names of future temporary VMs as keys. See example in config file.

Compilation of Danube Cloud platform

To build complete USB image, you need to compile the platform (based on slightly modified SmartOS).

# useradd -g 1 -s /bin/bash -d /data/compile -m compile
# usermod -P 'Primary Administrator' compile
# su - compile
$ git clone git://github.com/erigones/smartos-live
$ cd smartos-live/
$ git branch -av | grep new_release
$ git checkout new_release_20201105     # choose the newest one
$ ./configure
$ gmake live

This has enabled you to use make platform command from esdc-factory.

License

Copyright 2016-2020 Erigones, s. r. o.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this project except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

fishman/esdc-factory