debian-minimal-builder: A Shell repository from KeithIMyers

Debian ramdisk builder
----------------------

This directory contains an auto-build system for making a minimal Debian
system, suitable for including in a ramdisk.  It is tested and known to
build properly when built on a Debian system.

This project uses the travis-CI system to run test builds.  Travis uses
Ubuntu for its build systems and occasionally Ubuntu makes changes that are
incompatible with Debian and this project.  Thus the automated build might
fail - however for an error to be considered for fixing, it must be
reproducible on a Debian stable system.

It is intended to be simple to add additional configuration at both build
time and at runtime for specific use cases (see the Configuration sections,
below)

Usage:

    make build-depends          # install needed build packages

    make build/debian.stretch.armhf.cpio  # creates the root filesystem archive

After making sure the needed packages are installed, you just need to
run the full build.  The resulting image can be used to build a ramdisk
install - see the boards directory for various builders

Build time Configuration
------------------------

The various fixups and customisation that are applied to the basic Debian
installation can be seen in the packages.d directory (See the README
in that dir for more details on the layout of that directory).

To customise the system built for differing use cases, it is possible to
add a "config overlay".  This consists of a directory which optionally
extra configuration files to be used during the build.

To activate one or more config overlays, ensure that your environment has
exported a "CONFIGDIRS" variable - this variable is a list of the
directories to search for the overlays.  Configuration will be applied in
the order listed in this variable and the directories will be interpreted
relative to the builder directory - so use fully qualified pathnames to
ensure correctness.  Finally, the normal directory is only searched if it
is included in this list.

E.G:
To enable the normal processing and add one overlay after it:

    export CONFIGDIRS=". /tmp/buildoverlay"
    make build/debian.stretch.armhf.cpio

It is expected that this build system will be embedded in a larger system
and used as a tool - thus the larger build system will set the CONFIGDIRS
variable correctly.

Files that will be taken from the CONFIGDIRS:

authorized_keys
    This is a directory containing ssh public key files.  All files found will
    be copied into the built image (in the ~root/authorized_keys directory)
    before the customise phase scripts are run (Any files with duplicate names
    will only contain the last file found on the CONFIGDIRS path).
    Note that the public keys from the ssh_agent of the user creating the build
    will be automatically added as well.
    This directory can then be used by a customise phase script to create
    a real authorized keys file, or serve as a reference while using the built
    ramdisk system.
debian.multistrap.jinja
    This is the template of the multistrap config file.  If more than one of
    these files exists in the CONFIGDIRS then only the last defined one will
    be used.  This is a jinja2 template file.
apt.sources
    This subdirectory is searched for the deb sources .list file that defines
    the URL needed to download the main debian packages.
    To be able to pass additional settings into the multistrap template, lines
    of the form "##builder $variable $value" can be added to these files.
    This was specifically added to support defining the archive keyring
    package name, but could be used for other purposes.
packages.txt
    These files contain a list of additional packages to install in the
    buildroot.  All packages.txt files found in all the CONFIGDIRS will be
    appended to make the complete list of packages to install.  These are
    simply appeneded to the multistrap config file.
packages.d
    The files in these directories are used to apply changes to the installed
    buildroot.  See the README in the main packages.d file for the full list
    of what filed are loaded from here.
    All the package minimisation/fixup/add phases will look in every CONFIGDIR
    to find settings to apply - all matching files all the CONFIGDIRS will be
    applied.

Runtime Configuration
---------------------

During bootup - and before systemd starts - the scripts will scan all found
block devices looking for filesystems with a "/conf.d" directory in their
root and load any configuration found.  This script will not load any kernel
modules, so unless the block driver and filesystem have been statically
compiled, it will not find the filesystem.

These configuration files are simply tar.gz archives or sh scripts. First,
archives are extracted, in alphabetical order to /etc of the ramdisk, so files
in an archive towards the end of the list will overwrite those from an earlier
archive in case of a filename conflict. After all archives are extracted, scripts
are executed in alphabetical order. These steps occur before systemd starts so
systemd will only see the final (highest priority) configurations and after
effects of the scripts are applied.

For example:

    /conf.d/00-tomesh-base.tar.gz
            10-tomesh-wlan-mesh-top-gs07-rt5572.tar.gz
            11-tomesh-wlan-hostap-tplink-tl-wn722n.tar.gz
            50-node-config-save.tar.gz
            90-user-custom-configs.tar.gz
            bootstrap.sh

This allows local communities to customize nodes by distributing archives
of systemd.network configuration files, or any files that are to be read from
/etc. The user simply mounts the fat filesystem like a USB key and drops tar.gz
files into conf.d and can easily back them up across software updates.

If while running, you wish to save the current configuration, the
"config_save" script can be used to save the current /etc directory into an
archive called 50-node-config-save.tar.gz on the first block device found that
has a /conf.d directory.


TODO
----

A quick list of things that probably need doing:
(sorted vaguely by importance)

* Ensure ssh host keys are not saved in image and are regenerated if needed
* Add some build-time randomness into image
    dd if=/dev/urandom bs=512 count=1 of=$DEBOOT/var/lib/systemd/random-seed
* Load any extra randomness found on boot media
* Document the dependancy checking and its implications for deleted, renamed
  or new files
* Move some of these TODO items into the mesh-orange project
* Check all the available 802.11s flags and use the appropropriate ones
* use a known prefix for tunnel names to allow stable iptables rules and
  coexistance with other tunnel systems
* Write script for generating a basic config file to merge with image
* Use a better default access point name and PSK
* Use a better default root password
* Look for better options for the cjdns package
* look for a IPFS package
* u-boot rules to boot of one or the other of the initramfs images on fallback
* maybe add udev persistent-net-generator.rules - style network interface naming
* config storage - what about SPI flash?
KeithIMyers/debian-minimal-builder
