/chef

A systems integration framework, built to bring the benefits of configuration management to your entire infrastructure.

Primary LanguageRubyApache License 2.0Apache-2.0

Chef

Code Climate Build Status Master Build Status Master

Want to try Chef? Get started with learnchef

Chef is a configuration management tool designed to bring automation to your entire infrastructure.

This README focuses on developers who want to modify Chef source code. If you just want to use Chef, check out these resources:

Installing From Git

NOTE: Unless you have a specific reason to install from source (to try a new feature, contribute a patch, or run chef on an OS for which no package is available), you should head to the downloads page to get a prebuilt package.

Prerequisites

Install these via your platform's preferred method (apt, yum, ports, emerge, etc.):

  • git
  • C compiler, header files, etc. On Ubuntu/Debian, use the build-essential package.
  • ruby 2.3.0 or later
  • rubygems
  • bundler gem

Chef Installation

Then get the source and install it:

# Clone this repo
git clone https://github.com/chef/chef.git

# cd into the source tree
cd chef

# Install dependencies with bundler
bundle install

# Build a gem
bundle exec rake gem

# Install the gem you just built
gem install pkg/chef-VERSION.gem

Contributing/Development

Before working on the code, if you plan to contribute your changes, you need to read the Chef Contributions document.

The general development process is:

  1. Fork this repo and clone it to your workstation.
  2. Create a feature branch for your change.
  3. Write code and tests.
  4. Push your feature branch to github and open a pull request against master.

Once your repository is set up, you can start working on the code. We do utilize RSpec for test driven development, so you'll need to get a development environment running. Follow the above procedure ("Installing from Git") to get your local copy of the source running.

Reporting Issues

Issues can be reported by using GitHub Issues.

Full details on how to report issues can be found in the CONTRIBUTING doc.

Note that this repository is primarily for reporting chef-client issues. For reporting issues against other Chef projects, please look up the appropriate repository to report issues against in the Chef docs in the community contributions section. If you can't detemine the appropriate place to report an issue, then please open it against the repository you think best fits and it will be directed to the appropriate project.

Testing

We use RSpec for unit/spec tests. It is not necessary to start the development environment to run the specs--they are completely standalone.

# Run All the Tests
bundle exec rake spec

# Run a Single Test File
bundle exec rspec spec/PATH/TO/FILE_spec.rb

# Run a Subset of Tests
bundle exec rspec spec/PATH/TO/DIR

When you submit a pull request, we will automatically run the functional and unit tests in spec/functional/ and spec/unit/ respectively. These will be run on Ubuntu through Travis CI, and on Windows through AppVeyor. The status of these runs will be displayed with your pull request.

Building the Full Package

To build chef as a standalone package (with ruby and all dependent libraries included in a .deb, .rpm, .pkg or .msi), we use the omnibus system. Go to the omnibus README to find out how to build!

Updating Dependencies

If you want to change our constraints (change which packages and versions we accept in the chef), there are several places to do so:

Once you've made any changes you want, you have to update the lockfiles that actually drive the build:

  • To update chef's gem dependencies to the very latest versions available, run rake bundle:update.
  • To update chef's gem dependencies conservatively (changing as little as possible), run rake bundle:install.
  • To update specific gems only, run rake bundle:update[gem1 gem2 ...]
  • bundle update and bundle install will not work, on purpose: the rake task handles both the windows and non-windows lockfiles and updates them in sync.

To perform a full update of all dependencies in chef (including binary packages, tests and build system dependencies), run rake dependencies. This will update the Gemfile.lock, omnibus/Gemfile.lock, acceptance/Gemfile.lock, omnibus/Berksfile.lock, and omnibus_overrides.rb. It will also show you any outdated dependencies due to conflicting constraints. Some outdated dependencies are to be expected; it will inform you if any new ones appear that we don't know about, and tell you how to proceed.

How Chef Builds and Versions

Chef is an amalgam of many components. These components update all the time, necessitating new builds. This is an overview of the process of versioning, building and releasing Chef.

Chef Packages

Chef is distributed as packages for debian, rhel, ubuntu, windows, solaris, aix, and os x. It includes a large number of components from various sources, and these are versioned and maintained separately from chef project, which bundles them all together conveniently for the user.

These packages go through several milestones:

  • master: When code is checked in to master, the patch version of chef is bumped (e.g. 0.9.10 -> 0.9.11) and a build is kicked off automatically to create and test the packages in Chef's Jenkins cluster.
  • unstable: When a package is built, it enters the unstable channel. When all packages for all OS's have successfully built, the test phase is kicked off in Jenkins across all supported OS's. These builds are password-protected and generally only available to the test systems.
  • current: If the packages pass all the tests on all supported OS's, it is promoted as a unit to current, and is available via Chef's artifactory by running curl https://www.chef.io/chef/install.sh | sudo bash -s -- -c current -P chef
  • stable: Periodically, Chef will pick a release to "bless" for folks who would like a slower update schedule than "every time a build passes the tests." When this happens, it is manually promoted to stable and an announcement is sent to the list. It can be reached at https://downloads.chef.io or installed using the curl command without specifying -c current. Packages in stable are no longer available in current.

Additionally, periodically Chef will update the desired versions of chef components and check that in to master, triggering a new build with the updated components in it.

Automated Version Bumping

Whenever a change is checked in to master, the patch version of chef is bumped. To do this, the lita-versioner bot listens to github for merged PRs, and when it finds one, takes these actions:

  1. Bumps the patch version in lib/chef/version.rb (e.g. 0.9.14 -> 0.9.15).
  2. Runs rake bundle:install to update the Gemfile.lock to include the new version.
  3. Pushes to master and submits a new build to Chef's Jenkins cluster.

Bumping the minor version of Chef

After each "official" stable release we need to bump the minor version. To do this:

  1. Manually increment the minor version in the VERSION file that is in the root of this repo. and reset the patch version to 0. Assuming the current version is 12.10.57 you would edit VERSION to be 12.11.0.
  2. Run bundle exec rake version which will copy the version to the respective version.rb files in chef and chef-config.
  3. Run bundle exec rake bundle:install to update the base Gemfile.lock

Submit a PR with the changes made by the above.

Component Versions

Chef has two sorts of component: ruby components like berkshelf and test-kitchen, and binary components like openssl and even ruby itself.

In general, you can find all chef desired versions in the Gemfile and version_policy.rb files. The Gemfile.lock is the locked version of the Gemfile, and omnibus_overrides is the locked version of omnibus. build and test Gemfiles and Berksfile version the toolset we use to build and test.

Binary Components

The versions of binary components (as well as rubygems and bundler, which can't be versioned in a Gemfile) are stored in version_policy.rb (the OMNIBUS_OVERRIDES constant) and locked in omnibus_overrides. rake dependencies will update the bundler version, and the rest are be updated manually by Chef every so often.

These have software definitions either in omnibus/config/software or, more often, in the omnibus-software project.

Rubygems Components

Most of the actual front-facing software in chef is composed of ruby projects. berkshelf, test-kitchen and even chef itself are made of ruby gems. Chef uses the typical ruby way of controlling rubygems versions, the Gemfile. Specifically, the Gemfile at the top of chef repository governs the version of every single gem we install into chef package. It's a one-stop shop.

Our rubygems component versions are locked down with Gemfile.lock, and can be updated with rake dependencies.

There are three gems versioned outside the Gemfile: rubygems, bundler and chef. rubygems and bundler are in the RUBYGEMS_AT_LATEST_VERSION constant in version_policy.rb and locked in omnibus_overrides. They are kept up to date by rake dependencies.

Windows: Gemfile.lock is generated platform-agnostic, and then generated again for Windows. The one file has the solution for both Linux and Windows.

The tool we use to generate Windows-specific lockfiles on non-Windows machines is tasks/bin/bundle-platform, which takes the first argument and sets Gem.platforms, and then calls bundle with the remaining arguments.

Build Tooling Versions

Of special mention is the software we use to build omnibus itself. There are two distinct bits of code that control the versions of compilers, make, git, and other tools we use to build.

First, the Jenkins machines that run the build are configured entirely by the opscode-ci cookbook cookbook. They install most of the tools we use via build-essentials, and standardize the build environment so we can tear down and bring up builders at will. These machines are kept alive long-running, are periodically updated by Chef to the latest opscode-ci, omnibus and build-essentials cookbooks.

Second, the version of omnibus we use to build chef is governed by omnibus/Gemfile. When software definitions or the omnibus framework is updated, this is the file that drives whether we pick it up.

The omnibus tooling versions are locked down with omnibus/Gemfile.lock, and can be updated by running rake dependencies.

Test Versions

chef is tested by the chef-acceptance framework, which contains suites that are run on the Jenkins test machines. The definitions of the tests are in the acceptance directory. The version of chef-acceptance and test-kitchen, are governed by acceptance/Gemfile.

The test tooling versions are locked down with acceptance/Gemfile.lock, which can be updated by running rake dependencies.

The Build Process

The actual Chef build process is done with Omnibus, and has several general steps:

  1. bundle install from chef/Gemfile.lock
  2. Reinstall any gems that came from git or path using rake install
  3. appbundle chef, chef, test-kitchen and berkshelf
  4. Put miscellaneous powershell scripts and cleanup

Kicking Off The Build

The build is kicked off in Jenkins by running this on the machine (which is already the correct OS and already has the correct dependencies, loaded by the omnibus cookbook):

load-omnibus-toolchain.bat
cd chef/omnibus
bundle install
bundle exec omnibus build chef

This causes the chef project definition to load, which runs the chef-complete software definition, the primary software definition driving the whole build process. The reason we embed it all in a software definiton instead of the project is to take advantage of omnibus caching: omnibus will invalidate the entire project (and recompile ruby, openssl, and everything else) if you change anything at all in the project file. Not so with a software definition.

Installing the Gems

The primary build definition that installs the many Chef rubygems is software/chef.rb. This has dependencies on any binary libraries, ruby, rubygems and bundler. It has a lot of steps, so it uses a library to help reuse code and make it manageable to look at.

What it does:

  1. Depends on software defs for pre-cached gems (see "Gems and Caching" below).
  2. Installs all gems from the bundle:
    • Sets up a .bundle/config (code) with --retries=4, --jobs=1, --without=development,no_, and build.config.nokogiri to pass.
    • Sets up a common environment, standardizing the compilers and flags we use, in env.
    • Runs bundle install --verbose
  3. Reinstalls any gems that were installed via path:
    • Runs bundle list --paths to get the installed directories of all gems.
    • For each gem not installed in the main gem dir, runs rake install from the installed gem directory.
    • Deletes the bundler git cache and path- and git-installed gems from the build.
  4. Creates /opt/chef/Gemfile and /opt/chef/Gemfile.lock with the gems that were installed in the build.

Gems and Caching

Some gems take a super long time to install (particularly native-compiled ones such as nokogiri and dep-selector-libgecode) and do not change version very often. In order to avoid doing this work every time, we take advantage of omnibus caching by separating out these gems into their own software definitions. chef-gem-dep-selector-libgecode for example.

Each of these gems uses the config/files/chef-gem/build-chef-gem library to define itself. The name of the software definition itself indicates the .

We only create software definitions for long-running gems. Everything else is just installed in the chef software definition in a big bundle install catchall.

Most gems we just install in the single chef software definition.

The first thing

Appbundling

After the gems are installed, we appbundle them in chef-appbundle. This creates binstubs that use the bundle to pin the software .

During the process of appbundling, we update the gem's Gemfile to include the locks in the top level /opt/chef/Gemfile.lock, so we can guarantee they will never pick up things outside the build. We then run bundle lock to update the gem's Gemfile.lock, and bundle check to ensure all the gems are actually installed. The appbundler then uses these pins.

Other Cleanup

Finally, chef does several more steps including installing powershell scripts and shortcuts, and removing extra documentation to keep the build slim.

License

Chef - A configuration management system
Author: Adam Jacob (adam@chef.io)
Copyright: Copyright 2008-2016, Chef Software, Inc.
License: Apache License, Version 2.0 
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.