The fab is a product fabrication framework.
This version of Fab is actually the second generation framework, the first being referred to as oldfab.
The word "Fab" originates from the microelectronics industry. A fab, or fabrication plant is a factory where devices (eg. Integrated circuits) are manufactured for one of more customers. A fab is semantically connected to the most cutting edge technological factories in existence (Silicon chip foundries)
A fab is a very tightly controlled environment (clean room), but instead of keeping out physical impurities (e.g., dust and dirt), the Fab is used in fabricating systems while tightly controlling "logical" impurities (e.g., security threats, malware, etc.)
These release notes only contain a high-level overview, please refer to the design notes for detailed information, and help from the commands themselves.
The fab provides 'toolchain' utilities, which allows us to build products and collaborate on them using the same workflow and tools used on software projects.
Building is performed per-product, each in its own directory. We leverage 'make' to implement the 'build pipeline', git and covin for revision control and collaboration.
The output of a product is the product itself, and a recipe (very small footprint compared to the product) which can be use to automatically reproduce the product bit for bit.
The final product used by the end-user. The product is generated by formatting the "patched root"
The chroot'able root filesystem of a product:
- patched manually or automatically
- can be re-created automatically by applying the root patch as an overlay to the root component.
The chroot'able root filesystem of a product, built by applying the "root.spec" on the bootstrap.
The minimal chroot'able filesystem used to bootstrap the root, built from a "bootstrap.spec".
A set of (package name, package version tuples):
- a spec is created from a plan against a specific pool
- the same plan will generate different specs against different pools
A set of package names.
The plan from which the root.spec is created, by looking up the dependencies of listed packages recursively.
The plan from which we create the bootstrap spec, without recursively looking dependencies.
fab's product.mk is designed to be configurable and extendable with define based hooks and variables which should be set BEFORE including product.mk, because various elements are evaluated at include time (e.g., target prerequisites, variable exports):
<target>/pre # rules before default body (default: empty) <target>/post # rules after default body (default: empty) <target>/deps # override default dependencies for a rule <target>/deps/extra # extra dependencies for rule (default: empty) override built-in variables (e.g., ISOLABEL, CONF_SCRIPTS) include path/to/product.mk
Special exception - if you want to override built in built-in target rules for a target (I.e., when a /pre or /post hook isn't enough), you'll need to define them AFTER including the shared Makefile:
<target>/body # body of rules (default: defined, but can be overridden)
product.mk was originally based on pyproject-common's inheritable Makefiles pyproject.mk and debian-rules.mk. The main difference is that pyproject.mk is more flexible and has a better documented API (I.e., make help).
For example, you can not only inherit from product.mk in a Makefile, but you can also run product.mk as a standalone program which inherits variable definitions from its environment.
When you inherit from product.mk in a product Makefile, you can override built-in variables (but not built-in targets) before you include product.mk. This is because in product.mk built-in variables are assigned on the condition that they are not already set previously.
In a nutshell: define everything before including product.mk except for <target>/body overrides (which should be rarely needed - if ever).
You set the following defines BEFORE including the shared Makefile because target prerequisites are evaluated at include time:
<target>/pre # rules before default body (default: empty) <target>/post # rules after default body (default: empty) <target>/deps # override default dependencies for a rule <target>/deps/extra # extra dependencies for rule (default: empty)
Special case - if you want to override built in built-in rules for a target, you'll need to define them AFTER including the shared Makefile:
<target>/body # body of rules (default: defined, but can be overridden)
You can view which targets exist via the Makefile's embedded help target:
$ cd fab/contrib $ make -f product.mk help
I've extended fab and product.mk to support product configuration at build-time. This additional functionality is designed to address a few problems we've been having:
packages shouldn't perform product-specific configurations
For example, casper including scripts that configure users and such at boot time.
This violates separation of concerns and prevents packages from fulfilling their full utility as generic, reusable building blocks. It also increases the accidental complexity of the system by introducing unnecessary interdependencies.
Also, there is often significant overhead in changing a package to configure them to suit a specific product. This is especially true for stock packages, but generally creating multiple variants of a package just to support different configurations is inconvenient and time consuming.
boot time is not the correct time to perform product configuration
It doesn't scale, it lengthens the boot process and it limits the re-usability of casper.
support adjustments required for different releases with having to duplicate/fork a plan component
This is the primary reason the new fab design supports preprocessing of plans in the first place, in order to prevent the kind of inefficient and ugly duplication of product specifications (e.g., building blocks in "old" fab terminology) just to support minor adjustments.
build development/production variants of a product without having to modify the product (e.g., remove debug/development packages from the plan).
This allows the simultaneous development of both the "development" version of a product and the "production" version.
Previously the only way to affect product build was to use the "Makefile inheritance" to add pre/post hooks to targets, or even override the values of the target "body" and built-in variables.
Configuring a product this way is possible but relatively complex and inconvenient.
I have developed a couple of new powerful mechanisms to support more efficient product configuration:
- conf.d/ chroot scripts
- product configuration variables
- patches apply local or upstream patches
Any executable script in conf.d (default location, this can be changed) is copied into a temporary directory in root.patched (after the overlay, but before the removelist is applied) and executed while chrooted into root.patched. After execution the temporary directory is deleted.
Any type of script for which there is an interpreter in root.patched is supported (e.g., shell, perl, python). Static binaries are also supported but dynamic binaries are dangerous as differences in the library versions in the chroot may prevent the binary from running correctly, or more likely running at all.
The order of execution of scripts in conf.d depends on the script filename, so if have to control the order, you can append an integer (e.g., conf.d/10myscript).
The script is executed with arguments extracted from conf.d/args/<name>. By default, no arguments are passed. This supports re-usability of complex configuration scripts, but for simple configuration scripts it shouldn't be needed at all. Note that <name> in conf.d/args doesn't include priority prefixes, so you can change priority without having to rename conf.d/args/<name>.
Speaking of reusing complex scripts, just like rc*.d scripts, conf.d/ scripts can be symbolic links to shared scripts (e.g., /path/to/common-conf.d/<name>). Whether they should be is an entirely different question and the answer is usually no. Git supports symbolic links outside of a repository but a hardwired path will still be embedded in the product's repository, and you know how I feel about hardwired paths.
Pros of sharing configuration scripts:
- could be used to prevent duplication of logic in complex scripts (I.e., write once fix many times syndrome)
Cons:
- reduces readability: settings need to be separated to args/ or set in the environment, so its harder to glance at a script and see the whole picture.
- adds significant overhead: parsing of arguments, sanity checking, error messages, etc.
I think its usually preferable to put complex logic into a package and make the configuration script as simple as possible by calling the complex functionality it needs. If a configuration script has good enough primitives to leverage it can be made simple enough to resemble a configuration file itself. See skeleton conf.d scripts for an example.
In other words, I think its preferable to avoid sharing configuration scripts altogether, though I have supported and tested this capability in case we need it.
In case a single conf.d directory isn't enough, its possible to add additional directories by calling the run-conf-scripts macro in a pre/post hook, like this:
define root.patched/post
$(call run-conf-scripts, conf2.d)
endef
If instead of a directory of scripts you want to execute just a single script in a pre/post hook, thats also possible. Just call fab-chroot directly:
fab-chroot $O/root.patched --script path/to/script [args]
By setting the following in your product Makefile:
VAR1 = VALUE1 VAR2 = VALUE2 ... CONF_VARS = VAR1 VAR2 [ ... ]
You are describing a list of configuration variables that will effect:
- preprocessor definitions in fab-plan-resolve
- the environment of fab-chroot commands and scripts: the variables listed in CONF_VARS are exported into their environment.
Note: RELEASE is a mandatory built-in configuration variable. Its added by product.mk automatically, even if you don't define CONF_VARS at all. This is to ensure that common-plan components can depend on its existence to effect plan adjustments required for different releases (e.g., discover1 -> discover2).
For example:
$ cd skeleton $ cat Makefile RELEASE = rocky CONF_VARS = DEBUG DEBUG ?= y # empty string is false ifndef FAB_MAKEFILE_INCLUDE_PATH $(error FAB_MAKEFILE_INCLUDE_PATH not defined) else include $(FAB_MAKEFILE_INCLUDE_PATH)/product.mk endif $ cat plan/main #ifdef DEBUG #include <debug> #endif #include <boot> #include <console> #include <net>
Note that one things configuration variables don't effect are overlays, at least not by default. It is possible however to add this functionality by defining pre/post hooks which are effected by the value of the configuration variables.
Traditionally, TurnKey GNU/Linux has relied on upstream maintainers to include patches and updates in the packages they provide, and this is the preferred method. Sometimes, however, it is necessary to apply extra patches to packages. For example, a problem has been encountered that has been patched upstream, but the patch has not been included in Debian backports. In other cases, it may be useful to create a local patch to modify an appliance.
The command, fab-apply-patch, uses the patch command to apply a given patchfile to a chroot, typically build/root.patched.
# fab-apply-patch --help
Syntax: fab-apply-patch <patch> <path>
Apply patch on top of given path
Arguments:
<patch> Path to patch
<path> Path to apply patch ontop of (ie. build/root.patched)
Patches: Patches should be in unified context produced by diff -u
Filenames must be in absolute path format from the root
Patches may be uncompressed, compressed with gzip (.gz),
or bzip2 (.bz2)
fab-apply-patch has been designed to fail gracefully. If a patch cannot be applied, a warning is issued and the build is allowed to continue.
Patches should be in unified context produced by diff -u. Filenames must be in absolute path format from the root. Patches may be uncompressed, compressed with gzip (.gz), or bzip2 (.bz2).
Files that are patched are automatically backed up and marked <filename>.orig
If a patch fails to apply, a warning is issued. Patches that are no longer needed should be removed.
Patches placed in the patches.d directory will be applied to root.patched after the conf.d scripts are run and before the product-local removelist is applied. Patches are ordered by filename, so if you have to control the order, you can prepend an integer (e.g., patches.d/10myfix.patch)
Patches placed in the common/patches/turnkey.d directory will be applied to all appliances during the build after the common configuration scripts are run and before the common removelists are applied. This is recommended for patches to common code shared by all appliances. Note that patches placed in product/core/patches.d/ will only be applied to the core appliance build.
Patches that affect a group of appliances that share common code may be placed in common/patches/. For example, a patch that affects all the lamp appliances could be placed in common/patches/mylampfix.patch. In this case, the lamp.mk file must be modified to include the patchfile by name.
lamp.mk:
WEBMIN_FW_TCP_INCOMING = 22 80 443 12320 12321 12322 COMMON_OVERLAYS += apache adminer confconsole-lamp COMMON_CONF += phpsh apache-vhost adminer-apache adminer-mysql COMMON_PATCHES += mylampfix.patch include $(FAB_PATH)/common/mk/turnkey/php.mk include $(FAB_PATH)/common/mk/turnkey/mysql.mk