omnibus: A C repository from cparedes

Introduction

omnibus helps you build self-installing, full-stack software builds. For
example, we want to distribute the Chef client pre-bundled with its
dependencies: ruby, required gems, etc. The result is a single installable
shell-script that can be executed on the client side, installing Chef, and
configuring it.

omnibus starts with the idea of a “project”, which is comprised of a name,
version, and a build order. The build order references various “software”
packages, which are compiled and installed to a specific path on the
filesystem. (For the chef-full project, this would be /opt/opscode).

Using omnibus

To start using omnibus, you need a few dependencies:

Java – either the Sun JDK, or the Open JDK
Leiningen
Your operating systems base build utilities – gcc, autoconf, etc.

To install Leiningen, follow the instructions here: http://github.com/technomancy/leiningen

Once you have Leiningen installed, run:

$ lein deps

From the top of the omnibus source tree.

To start a build:
(make sure the project includes the ‘prep’ step which will create the appropriate directories)

$ lein repl
omnibus.core=> (build-fat-binary “chef-full”)

The results will be in ./pkg when the build is complete.

Adding software to omnibus

To add new software to omnibus, you want to extract it and commit the source in the ‘./source’ directory. Then add a configuration to config/software. For example, to build gdbm:

1. Download the source for gdbm-1.8.3 2. Unpacked it in ./source 3. Removed the tarball 4. Added the following configuration in config/software/gdbm.clj


(software "gdbm" :source "gdbm-1.8.3"
                 :steps [["./configure" "--prefix=/opt/opscode/embedded"]
                         (if (is-os? "darwin") ["perl" "-pi" "-e" "s/BINOWN = bin/BINOWN = root/g" "Makefile"])
                         (if (is-os? "darwin") ["perl" "-pi" "-e" "s/BINGRP = bin/BINGRP = wheel/g" "Makefile"])
                         ["make"]
                         ["sudo" "make" "install"]])

The :source field should be the name of the directory created by unpacking the tarball. The steps are a vector of vectors, each of which corresponds to a build step. These steps are passed directly to clojure.java.sh/sh, which will execute them directly. You can use two handy functions to make the build be different based on operating system or machine architecture: is-os? and is-machine?.

Adding software to a project

In config/projects, you will find the actual project definitions themselves. These contain a name for the project, a version number, and the build order. For the chef client:

(project 
  "chef-full" 
  "0.9.8" 
  [ "zlib" "libiconv" "db" "gdbm" "ncurses" "openssl" "libxml2" "libxslt" "ree" "rubygems" "rsync" "chef" ])

The build order will run in order.

Enjoy!

The full build process

If you want to kick off a full build, the easiest way is with:

$ sudo lein run --project-name "chef-full" --bucket-name FULL_STACK --s3-access-key "ACCESS_KEY" --s3-secret-key "SECRET_KEY"

The resulting build artifact will be dropped off in the S3 bucket named above. If you want to submit a build to us, drop it off in your own bucket, and let us know where and how to test the final product.

Operating System Prep

Fedora 15

yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo which
gem install fpm ohai --no-rdoc --no-ri
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

SL 6

yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo
gem install fpm ohai --no-rdoc --no-ri
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

CentOS 5

wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
yum update -y
yum install -y gcc gcc-c++ kernel-devel make autoconf flex bison git java-1.6.0-openjdk java-1.6.0-openjdk-devel ruby ruby-libs ruby-devel ruby-docs ruby-ri ruby-irb ruby-rdoc ruby-mode rubygems rpm-build libxml2 libxml2-devel libxslt libxslt-devel wget automake help2man libtool gettext texinfo
gem install fpm ohai --no-rdoc --no-ri
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Debian 6

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Ubuntu 10.04

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext texinfo
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Ubuntu 11.04

apt-get update -y
apt-get -y install build-essential binutils-doc autoconf flex bison git-core openjdk-6-jdk default-jdk ruby ruby1.8 ruby1.8-dev rdoc1.8 irb1.8 ri1.8 libopenssl-ruby1.8 rubygems libtool dpkg-dev libxml2 libxml2-dev libxslt1.1 libxslt1-dev help2man gettext texinfo
update-java-alternatives -s java-6-openjdk
gem install fpm ohai --no-rdoc --no-ri
ln -s /var/lib/gems/1.8/bin/* /usr/local/bin
wget https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Joyent Smartmachine

pkgin up
pkgin in autoconf-2.68 gtexinfo bison flex
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

OpenIndiana 148

#!/bin/bash
# Notes
# pkg:/developer/gcc-3 is required over gcc-43, as Ruby was built with gcc3.
# Ruby build requires the same compiler to build extensions as was used to build
# Ruby itself.
# Need to use /opt/opscode/embedded/bin/autoconf?
# ln -s /usr/bin/automake-110 /usr/bin/automake
pkg install pkg:/developer/build/autoconf \
pkg:/developer/build/automake-110 \
pkg:/developer/build/gnu-make \
pkg:/developer/build/libtool \
pkg:/developer/build/make \
pkg:/developer/gcc-3
pkg:/developer/java/jdk \
pkg:/developer/lexer/flex \
pkg:/developer/library/lint \
pkg:/developer/linker \
pkg:/developer/macro/cpp \
pkg:/developer/object-file \
pkg:/developer/parser/bison \
pkg:/developer/versioning/git \
pkg:/file/gnu-coreutils \
pkg:/library/libxml2 \
pkg:/library/libxslt \
pkg:/ruby-dev \
pkg:/runtime/ruby-18 \
pkg:/system/header \
pkg:/system/library/math \
pkg:/system/library/math/header-math \
pkg:/system/xopen/xcu4 \
pkg:/text/gnu-gettext \
pkg:/text/texinfo \
pkg:/web/wget \
pkg:/compatibility/ucb \
pkg:/text/gnu-sed
gem update
gem install json_pure
mv /var/ruby/1.8/gem_home/gems/json-1.4.6 /tmp ; ln -s /var/ruby/1.8/gem_home/gems/json-1.5.1 /var/ruby/1.8/gem_home/gems/json-1.4.6
gem install ohai -v=0.5.8
gem install fpm

FreeBSD 8.2

portsnap fetch update
cd /usr/ports/ports-mgmt/portupgrade && make && make install # Be sure that BATCH=yes
portupgrade -a -m BATCH=yes # Upgrade all ports
portinstall lang/python26 devel/autoconf textproc/flex devel/bison devel/git java/openjdk6 lang/ruby19 devel/gmake
gem update
gem install ohai
gem install fpm
wget --no-check-certificate https://github.com/technomancy/leiningen/raw/stable/bin/lein && mv lein /usr/local/bin && chmod 755 /usr/local/bin/lein
git clone git://github.com/adamhjk/omnibus.git
cd omnibus
mkdir build
lein deps

Bundled software and licensing

All software under the “source” directory retains the license by which it was originally distributed.

cparedes/omnibus