GUB the Grand Unified Builder • Home • Basics • Applications • History • Links GUB -- Grand Unified Builder WHAT IS GUB Grand Unified Builder (GUB, http://lilypond.org/gub) is a mini source-based distribution and packaging system. It cross-compiles several packages with their dependencies and assembles them into a single installation package. GUB aims at correctness ie, be dependable and reproducible, while avoiding gratituous rebuilds. It was originally developed for turnkey building of all LilyPond binary installers. Currently, GUB supports GNU/Linux, FreeBSD, MacOS-X and Microsoft Windows installation packages. All packages are fully relocatable and can be installed with user privileges. GUB uses plain Python classes to describe package build specifics. For reasonably well behaved packages, individual build specifications can stay very simple by extending the generic build specification classes. USAGE Run these programs with --help argument for instructions bin/gpkg - Gub Package Manager bin/gub - the Gub Universal Builder bin/gib - Gub (platform dependent) Installer Builder bin/gub-tester - build tester; integrates with Bazaar,CVS,Darcs,Git,SVN and gub bin/gupdate - check for and show or update new upstream releases bin/cygwin-packager - roll cygwin package BUGS Send bug reports and patches to lilypond-devel@gnu.org. Supported installer packages GNU Denemo GNU LilyPond with all essential dependencies. Currently, 207 package build specifications are included in GUB. Supported installer platforms darwin-ppc tar.bz2 file for Darwin 7 (MacOS 10.3)/PowerPC darwin-x86 tar.bz2 file for Darwin 8 (MacOS 10.4)/x86 mingw mingw executable for Windows32 linux-x86 shar archive for Linux/x86 linux-64 shar archive for Linux/x86_64 linux-ppc shar archive for Linux/PowerPC freebsd-x86 shar archive for FreeBSD 6/x86 freebsd-64 shar archive for FreeBSD 6/x86_64 cygwin .tar.bz2 packages for Cygwin/Windows32 arm shar archive for Linux/ARM (largely untested) debian shar archive for Debian (largely untested) LICENSE GUB is distributed under the terms of the GNU General Public License, version 2 or later. Gub comes with NO WARRANTY. ACQUIRING The Git repository is at https://github.com/gperciva/gub You can get it by doing git clone git://github.com/gperciva/gub.git or download the latest tarball doing wget https://github.com/gperciva/gub/tarball/master GUB home page http://lilypond.org/gub INSTALLING * You need - about 9 GB of free space (for all platforms) - standard unix shell utilities: cat, cp, install, mv, rm, sed, ... - Python 2.4 or newer (2.5, 2.6, 3.0 are known to work) - a standard unix development environment with GCC and G++, including: - 32-bit and 64-bit libraries - the file(1) utility * Try bin/gub --help make help Examples * Prepare a cross build environment for darwin-x86 bin/gub darwin-x86::cross/gcc * Cross build bison for FreeBSD x86_64 bin/gub freebsd-64::ftp://ftp.gnu.org/pub/gnu/bison/bison-2.3.tar.gz * Make a LilyPond installer for your platform bin/gub lilypond-installer this leaves the installer in uploads/ * Cross build Denemo (http://denemo.org) for Windows (mingw32) make denemo this leaves the Windows installer in uploads/ GUB the Grand Unified Builder • Home • Basics • Applications • History • Links GUB BASICS ROADMAP directory contents and use . (GNUmakefile, *.make) make drivers (try: make help) bin/ programs (gub, gpkg, gib, ...; try: bin/gub --help) gub/ python library gub/specs/ build specifications (gub/specs/ hello.py,bison.py,denemo.py,lilypond.py...) patches/ patch files used in build specifications target/ build trees log/ (gub.log, ...) (build) log files uploads/ installers target/<platform>/ working tree for <platform> target/linux-x86/src/ linux-x86 unpacked sources target/linux-64/build/ linux-64 build output target/darwin-x86/log/ darwin-x86 most recent lilypond build log lilypond.log target/mingw/packages/ .gup packages for mingw target/darwin-ppc/log/ status of partial darwin-ppc builds status/ target/freebsd-x86/root/ freebsd-x86 build and install root target/tools/root/usr/ native tools used for build bin/ sourcefiles/ non-patch files used in build specifications nsis/ nsis windows installer scripts BASIC WORKINGS Checksums and rebuilds GUB seeks to minimise unnecessary rebuilds while being strict about important changes. Before starting the actual building of a package, the build recipe is executed as a dry-run and a checksum is calculated from these build commands. This checksum includes the environment variables DISPLAY, FTP_PROXY, GUB_TOOLS_PREFIX, HOME, HTTP_PROXY, LIBRESTRICT, PATH [no other environment variables are allowed through]. The packages' build checksum is matched with the previous build and if it changes, the package is rebuilt. Otherwise, only the binary package is reinstalled, if necessary. This means that the .py build specs can be changed and refactored at will and as long as no actual build commands change, nothing will be rebuilt. This makes that forcing rebuilds can can be tricky: gub refuses to do so if it deems rebuilding unnecessary. SHARING DIRECTORIES GUB uses an ABI environment variable to work around some build bugs in some packages. This variable is not checked by the environment-changed function, so problems will probably occur if you share your GUB build directory between a 32-bit and 64-bit OS. If you change your operating system, delete the target/ directory of GUB before trying to build it again. LIBRESTRICT A common problem with [making reproducible cross] builds is using files or features that are not controlled by the [GUB] build process, but just happen to be on the build host and happen to differ between most build hosts. GUB uses the LD_PRELOAD feature with the gub/specs/librestrict.py package and code from ./librestrict/restrict.c to disallow reading from build host files. This makes for much better reprocucibility, especially between different build hosts. The downside to this LIBRESTRICT feature is that quit a few packages are not prepared for cross building. LIBRESTRICT will catch these errors and they have to be fixed in the package's spec file. This is what such a LIBRESTRICT assertion could like librestrict:error:/home/janneke/vc/gub/target/mingw/root/usr/cross/libexec/gcc/i686-mingw32/4.1.1/cc1plus: tried to open () file /home/janneke/vc/lilypond/lily/main.cc librestrict:allowed: /home/janneke/vc/gub/target/mingw /tmp /dev/null /dev/urandom /proc/self cc1plus: internal compiler error: Aborted Please submit a full bug report, with preprocessed source if appropriate. See for instructions. [1]10:42:08 janneke@peder:~/vc/gub $ you often find these in config.log files when package-dependant code blondly looks in /usr/include or /usr/lib. stat-restriction LIBRESTRICT has an experimental strict mode enabled by setting the environment variable LIBRESTRICT=open:stat This will even disallow any STATting outside the target's build tree. It would be nice to get this to work and some effort has been done. However, this is potentially a lot of work as not only upstream packages but also build system authors [autoconf, automake, libtool etc.] look at the build host root file system while cross building. Simplifying package builds or avoiding code duplication Another idea of GUB is to identify, fix and generalise build quircks and move them from a package's spec file into one of the generic build classes. This increases maintainability by avoiding code duplication, makes spec files smaller, simpler and also increases the chances of a new package's spec file to only need a source url to specify the build. BASIC USAGE Build package hello for platform linux-x86 bin/gub linux-x86::hello Examine why package hello wants to rebuild less target/linux-x86/log/build.log search from end back to `mismatch:' [type: FC-c?mismatch:RET] Force a rebuild of package hello for platform linux-x86 rm -rf target/linux-x86/*/hello-* Force a rebuild of package hello for all platforms rm -rf target/*/*/hello-* Prepare to rebuild entire linux-x86 platform rm -rf target/linux-x86 Prepare to rebuild everything, all platforms rm -rf target SPEC FILES Plain url builds <--no spec--> Well behaved packages can be built without spec file, straight from an url. For example, to build bison-2.2 as a build tool, do bin/gub tools::http://ftp.gnu.org/pub/gnu/bison/bison-2.2.tar.gz This produces target/tools/packages/bison-2.2.tools.gup target/tools/packages/bison.tools.hdr target/tools/packages/bison.checksum Check files installed under target/tools/root/ by doing bin/gpkg -p tools files bison To build the very latest findutils for your architecture, do bin/gub git://git.savannah.gnu.org/findutils.git Simple spec gub/specs/hello.py from gub import target class Hello (target.AutoBuild): source = 'http://lilypond.org/downloads/gub-sources/hello-1.0.tar.gz' build it for mingw by doing bin/gub mingw::hello __TOOLS package with dependencies gub/specs/autoconf.py from gub import tools class Autoconf__tools (tools.AutoBuild): source = 'http://ftp.gnu.org/pub/gnu/autoconf/autoconf-2.63.tar.gz' parallel_build_broken = True dependencies = [ 'm4', 'perl', ] build it by doing bin/gub tools::autoconf configure flags gub/specs/icoutils.py from gub import tools class Icoutils__tools (tools.AutoBuild): dependencies = ['libpng-devel'] configure_flags = (tools.AutoBuild.configure_flags + ' --with-libintl-prefix=%(system_prefix)s' + ' --disable-nls') __<platform>__<arch>-specific instructions For platform specific instructions, append "__<platform>" to the class name. For platform+architecture-specific instructions, "__<platform>__<arch>" to the class name. gub/specs/pixman.py from gub import target class Glib (target.AutoBuild): ... class Glib__freebsd (Glib): dependencies = Glib.dependencies + ['libiconv-devel'] configure_variables = Glib.configure_variables + ' CFLAGS=-pthread' class Glib__freebsd__x86 (Glib__freebsd): # Must include -pthread in lib flags, because our most beloved # libtool (2.2.6a) thinks it knows best and blondly strips -pthread # if it thinks it's a compile flag. # FIXME: should add fixup to update_libtool () compile_flags = ' G_THREAD_LIBS=-pthread G_THREAD_LIBS_FOR_GTHREAD=-pthread ' String expansion In all build commands, standard Python string expansion %(string-name)s can be used. These strings are taken from class Settings in gub/settings.py and from the package's build class and its parent classes. Adding string expansion variables to a spec build class can be done in several ways • class variable class Foo (target.Autobuild): common_configure_flags = ' --without-java' • instance variable class Foo__mingw (Foo): def __init__ (self, settings, source): Foo.__init__ (self, settings, source) self.target_gcc_flags = '-mms-bitfields' • decorated funtion returning a string class Bar__mingw (target.Autobuild): @context.subst_method def target_os (self): return 'windows' • Manipulating the substitution_dict using gub/build.py's class Change_dict or overriding the get_substitution_dict method. Pre-defined strings Typical variables defined by settings are build_architecture=x86_64-linux build_os=linux build_platform=linux-64 platform=target target_architecture=x86_64-linux target_bits=64 target_cpu=x86_64 target_os=linux target_platform=target Typical variables defined by the build class compile_flags= builddir=.../target/build/... config_cache_overrides= configure_binary= .../configure configure_command= sh .../configure ... configure_flags= --prefix=... --enable-shared --enable-static configure_variables= CFLAGS=-I.../target/root/usr/include install_command= install_flags= install_prefix= install_root= make_flags= srcdir=.../target/src/... More examples See gub/specs/*.py for some more examples. GUB -- LilyPond INSTALLING Get GUB git clone git://github.com/gperciva/gub.git or wget https://github.com/gperciva/gub/tarball/master In addition to the requirements for GUB, you need • A TeX installation • texinfo • zip Build everything make lilypond this leaves installers and documentation in uploads/lilypond-VERSION-RELEASE.darwin-ppc.tar.bz2 uploads/lilypond-VERSION-RELEASE.darwin-x86.tar.bz2 uploads/lilypond-VERSION-RELEASE.freebsd-64.shar.sh uploads/lilypond-VERSION-RELEASE.freebsd-x86.shar.sh uploads/lilypond-VERSION-RELEASE.linux-64.shar.sh uploads/lilypond-VERSION-RELEASE.linux-ppc.shar.sh uploads/lilypond-VERSION-RELEASE.linux-x86.shar.sh uploads/lilypond-VERSION-RELEASE.mingw.exe uploads/lilypond-VERSION-RELEASE.documentation.tar.bz2 uploads/lilypond-VERSION-RELEASE.webdoc.tar.bz2 uploads/webdoc/VERSION uploads/lilypond-VERSION-RELEASE.test-output.tar.bz2 uploads/wbtest/VERSION More examples * Make a LilyPond installer for your platform bin/gub lilypond-installer OR (old style) bin/gub lilypond # package with dependencies bin/gib --branch=lilypond=git.sv.gnu.org--lilypond.git-master lilypond #installer * Build all LilyPond installers make lilypond this leaves installers for GNU/Linux (x86, x86_64, powerpc), Windows, MacOSX (x86 and powerpc), FreeBSD (x86, x86_64) and documentation in uploads/ * Optional: Prepare build environment make -f lilypond.make bootstrap TESTING The test-lily directory contains scripts for running various tests. Run without arguments for instructions cron-builder.py - entry point from continuous crontab testing test-binary.py - test binary builds with-lock.py - run a command with a lockfile rsync-lily-doc.py - prepare documentation for upload dist-check.py - check tarball for anomalies cygwin-packager.py - repackage GUB packages into Cygwin tarballs. CRON USAGE I have the following in my crontab • Hourly build of LilyPond subpackage for all platforms except native (native is darwin-pcc?) test-lily/cron-builder.py --package\ --test-options "--to lilypond-cvs@gnu.org --quiet --from hanwen@lilypond.org --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\ darwin-x86 freebsd-x86 freebsd-64 mingw linux-x86 linux-64 • Hourly build of native installer, to run documentation and dist check. test-lily/cron-builder.py\ --package --installer --tarball --docs\ --test-options "--to lilypond-cvs@gnu.org --dependent --quiet\ --from hanwen@lilypond.org --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\ --unversioned\ darwin-ppc • Nightly build of all installers, to be ready for shipping binaries in the morning. test-lily/cron-builder.py\ --installer\ --test-options "--to lilypond-cvs@gnu.org --quiet --from hanwen@lilypond.org\ --smtp smtp.xs4all.nl --repository downloads/lilypond.git"\ darwin-x86 freebsd-x86 freebsd-64 mingw linux-x86 linux-64 darwin-ppc GUB -- GNU Denemo INSTALLING Get GUB git clone git://github.com/gperciva/gub.git or wget https://github.com/gperciva/gub/tarball/master Build everything make denemo this produces a windows installer in uploads/denemo-x.y.z-b.mingw.exe More examples * Build main Denemo installers make PLATFORMS='mingw linux-x86 linux-64' denemo * Build all Denemo installers make 'PLATFORMS=mingw linux-x86 linux-64 linux-ppc darwin-ppc darwin-x86 freebsd-x86 freebsd-64' denemo this leaves installers for GNU/Linux (x86, x86_64, powerpc), FreeBSD (x86, x86_64) and Windows, and documentation in uploads/ OR (old style) bin/gub denemo bin/gib denemo --branch=lilypond=git.sv.gnu.org--lilypond.git-master --branch=denemo=git.savannah.gnu.org--denemo.git-master * Optional: Prepare build environment make -f lilypond.make bootstrap GUB the Grand Unified Builder • Home • Basics • Applications • History • Links GUB -- Grand Unified Builder -- HISTORY HISTORY The story starts June 1999 with a crazy guy with an itch to run LilyPond on Windows. To get a feel for the times, this was LilyPond-1.1.47, requiring Egcs 1.1, Python 1.5, Guile 1.3, discussing on help-gnu-music@gnu.org. Two years later Cygwin cross, our first cross build environment is released. This is evolved into a patch set for mknetrel. Spring 2005: In a bold move a set of patches for MinGW are created to build LilyPond and all its [17] dependencies. Simultaniously, a cross build environment and installer for MacOSX is created. Creating a cross build environment is not too difficult, esp. not if you know how to use Google. The problem is that they always break and never work on other machines. GUB starts as an effort to unify the Windows and MacOS builders and to make the process somewhat dependable and reproducible, i.e.: even more free time for the developers (if there is such a thing) as the builds always succeed. It uses Python instead of make and bash and adds checksums for the package build specs. The unification of cross builds greatly facilitates adding of new architectures and before long GUB supports cross building lilypond binaries for 9 different platforms. GUB2 adds serialization of the Python build code before execution. This makes the build checksumming less coarse, ie, less gratuitous rebuilds and more reliable. Also, all Python code is executed before the build process starts, which means that there is less chance of silly run time errors. Lots of packages are added in an effort to replace the OpenEmbedded-based build system for the Samco 3G video phone at SES, Philips Research Eindhoven. In Novell's Hackweek-iii GUB enters into its Third Life with GUB3, adding cross-platform build dependencies and patches for an OpenOffice.org MinGW cross build, including all essential libraries. Later, Inkscape is added, exploring and implementing x.org dependencies. After that, building a windows installer for GNU Denemo took only one day, including all dependencies. GUB the Grand Unified Builder • Home • Basics • Applications • History • Links Links autopackage.org openSUSE build service codesourcery.com pre-GUB LilyPond installers emdebian.org GUB build specifications GUB patches lilypond.org/blog/janneke/gub3-ooo-mingw-cross-build openembedded.org scratchbox.org Scratbox2; sb2 Dan Kegel's crosstool