Ansible Powerplay

Powerplay allows you to run multiple Ansible playbooks in parallel. Depending on how you organize your playbooks, this can be a solid win. I basically before this had been doing a playbook with multiple includes for other playbooks representing different servers in our stack. Playbook launching of playbooks is slow and very serial.

Basically, the playbooks are all contained, so no interdependencies. And in my case, running in the cloud, so no reason why they can’t be running in parallel

Powerplay allows you to specify vars common to all playbooks, and also vars specific to some playbooks so by which you can make your setup very DRY.

All the Ansible playbooks are executed in seperate processes, and thus avoiding a number of the “side effects” you would normally encounter with running multiple playbooks with Ansible includes.

For example, here is Powerplay integrated with tmux:

Release Notes

Please see Release Notes.

Hilights

DSL

The version 1.x releases adds new features to the DSL, most notably, nestable groups, and being able to label each group as :sync or :async.

STDOUT from ansible-playbook

The capture of the output from ansible-powerplay is handled a bit more intelligently. If you do not specify the –tmux (or -m) option, all output is now currently captured by Powerplay and redumped to the console.

Because you may still want to see the color from ansible-powerplay, you can alter ansible.cfg and add the following line:

force_color = 1

Please see New STDOUT capturing with 1.0.x

Group Sequence Numers

We now allow groups to have sequence numbers, as of 1.1. Basically, if you specify a sequence, the variable you designate will be assigned a value in each of the sequence, with the group re-excuted. for example:

group :first, "async group with sequencing",
            seq: { iter: [1, 5, 9, :dodo] } do
  book :nat, "nat.yml"
  book :dat, "dat.yml"
  book :rat, "rat.yml"
end

as you can see (in the development.play sample) the variable “iter” will be successively assigned the element in the [] array, with the underlying playbooks called. The is the function equivalent of the following:

group :first, "async group without sequencing" do
  configuration do
    iter 1
  end
  book :nat, "nat.yml"
  book :dat, "dat.yml"
  book :rat, "rat.yml"
end

group :first, "async group without sequencing" do
  configuration do
    iter 5
  end
  book :nat, "nat.yml"
  book :dat, "dat.yml"
  book :rat, "rat.yml"
end     

group :first, "async group without sequencing" do
  configuration do
    iter 9
  end
  book :nat, "nat.yml"
  book :dat, "dat.yml"
  book :rat, "rat.yml"
end

group :first, "async group without sequencing" do
  configuration do
    iter :dodo
  end
  book :nat, "nat.yml"
  book :dat, "dat.yml"
  book :rat, "rat.yml"
end

As you can see, the new sequencing can be quite succinct.

Features and Cavets

Integration with TMUX

When running multiple Ansible Playbooks concurrently, one would like to be able to see the output of each in a reasonable manner. To faciliate this in this initial realse, we shall make heavy use of TMUX panes to dump the output.

So basically, you need as many panes as you have concurrent Ansible Playbooks in this initial release. In subsequent releases, Curses will be directly leveraged to create “tabs” for the multiple output streams. We may even do this, still, through TMUX.

Your input on this is strongly encouarged. We will not be supporting Screen at all. Sorry.

New STDOUT capturing with 1.0.x

The new capture, while properly capturing the STDOUT of concurrent async runs, does not display until ansible-playbook completes. If you are like me, you’ll like to see the progress as it runs. You still can using tmux with the –tmux (-m) option.

DSL Terminology & Documentation

Note that this is the DSL for version 1.x of PowerPlay. For 0.x, please see those tags in GitHub.

DSL

The DSL is straightforward as possible, simple and elegant to allow you to write your Powerplays in a DRY manner.

For examples, please see the following:

stack.play	This is loaded by default, and you must be in your current directory
development.play	This is a fullblown Power Playbook for a hypothetical development stack.
production.play	This is a fullblown Power Playbook for a hypothetical production stack.
playbooks	Sample Ansible playbooks called by Powerplay.

To run the powerplay example:

Install Ansible Powerplay
- gem install ansible-powerplay
Clone this project locally, then cd into the examples directory
- git clone https://github.com/flajann2/ansible-powerplay.git
- cd ansible-powerplay/examples
source ansible-paths and run Powerplay
- source ansible-paths.sh
- powerplay play -p development -v2

Note that I deliberately left a missing “elasticsearch.yml” so you can see how Powerplay handles the errors.

configuration

You can intersperse configuration blocks anywhere, and the expected nested scoping will take effect.

playbooks

playbooks are a collection of groups, and a group defaults to async mode for its members.

Group are normally executed serially. This will allow you to organize your plays in an intelligent manner to deploy and manage resources and assets that may have to be done in a serial manner.

group

A group is a collection of books or other groups that all execute in parallel by default. Books are required to be independent of each other. If they are not, you can set them up to execute serially.

book

A book has a direct correspondence to an Ansible playbook, and will execute that Yaml file given the configuration variables as parameters.

Here is where var inheritance becomes useful. Note that all the configuration variables set at the time the book is called are all passed in as –extra-vars to Ansible Playbook. The Playbook may not need all the vars passed in, but care must be taken that no vars are used in a different manner than expected. We currently have no way of knowing which vars are needed or not, and to specifiy that would make the syntax messy and loose some of the advantages of var inheritance.

Installation

Easy installation. From command-line:

gem install ansible-powerplay

Or from a gemfile:

gem 'ansible-powerplay'

Use

Basically, cd to the root of your Ansible directory, and a .play file (see the example at: stack.play.)

You can place a config clause either globally, inside of playbooks, inside of groups, and the variable set up this way are inherited to the inner clauses, thus allowing you to keep your specifications DRYer.

For example:

# This is a global system configuration
configuration :system do
  playbook_directory "playbooks"
end

Note that ‘playbook_directory’ is special, as it allows you to define the directory all of your Ansible playbooks can be found. You can also specify this anywhere you can use the configuration clause, so you may set up different playbook directories for different playbook collections.

# sṕecific configuration for :development
configuration do 
 stack :development
 krell_type "t2.small"
 servers 1
 rolling 3
 krell_disk_size 20
end

The above shows Ansible variables for my specialiezd setup that is geared with work with AWS. You are free to specify any variables here, which will be injected into ansible-playbook through the ‘–extra-vars’ parameter.

Here is a group clause with a single book in it:

# Groups are executed serially.
group :first, "our very first group" do
  # Books within a group are executed in parallel,
  # and therefore must be independent of each other.
  book :nat, "nat.yml"
end

Which issues the following command to Ansible (based on the earlier configuration):

ansible-playbook playbooks/nat.yml \
  --extra-vars "playbook_directory=playbooks stack=development krell_type=t2.small servers=1 rolling=3 krell_disk_size=20"

And if our group had more book entries, as in the second example:

group :second, "our second group" do
  book :rabbit, "rabbitmq_cluster.yml" do
    krell_type "t2.medium"
  end

  book :es_cluster, "elasticsearch_cluster.yml" do
    esver "1.7.4"
    cluster_name :es
    servers 3
    heapsize "2g"
    krell_type "t2.medium"
    krell_disk_size 200
  end
end

Both the :rabbit and :es_cluster books would be executed in parallel.

Dividing up your specs in other PowerPlay files

Ruby, the underlying language, give you a lot of things for “free”, like allowing you to load other powerplay files, for example:

load 'production.play'

We mention this here for those who may not be familiar with Ruby, but may wish to section off your specifications thusly.

You don’t really need to know any Ruby, but it could increase the span of what you might want to do. To get a quick taste, please checkout Ruby in 20 Minutes.

It is also possible to leverage Ruby’s metaprogramming techniques to create templates for your specificaitons, but at some point, as time allows, I may directly support this in the DSL. Please let your wishes be known to me for this and any other feature you might want to see.

Running Powerplay

If you type ‘powerplay’ without parameters, you are greeted with:

Commands:
  powerplay help [COMMAND]                                            # Describe available commands or one specific command
  powerplay play <script> -p, --play=[NAME|all] Which playbook shelf  # Run the powerplay script.
  powerplay ttys                                                      # list all the TMUX ptys on the current window.

Options:
  -v, [--verbose=[1|2|3]]
                           # Default: 0

Please use the help feature to explain the subcommands and options. We shall be adding many more subcommands and options as our needs demands. If you like to see something here, please submit it as an issue on Github.

And for an example of play help, (note that this may not be up-to-date, so please run ‘powerplay help play’ on your installe version!)

powerplay help play
Usage:
  powerplay play [script] -p, --play, --power, --play=[NAME[ NAME2...]|all]

Options:
  -m, [--tmux=[WINDOWNUMBERopt]]                                                                                              #  Send output to all tmux panes in the current window, or the numeric window specified.
  -p, --play, --power, --play=[NAME[ NAME2...]|all]                                                                           # Which PowerPlay playbooks (as opposed to Ansible playbooks) to specifically execute.
  -g, [--group=[NAME[ NAME2...]|all]]                                                                                         #  Which groups to execute.
                                                                                                                              # Default: [:all]
  -c, [--congroups], [--no-congroups]                                                                                         # Run the groups themselves concurrently
  -b, [--book=[NAME[ NAME2...]|all]]                                                                                          # Which books to execute.
                                                                                                                              # Default: [:all]
  -u, [--dryrun], [--no-dryrun]                                                                                               # Dry run, do not actually execute.
  -x, --extra-vars, [--extra=<BOOKNAME|all>:"key1a=value1a key2a=value2a... " [BOOKNAME2:"key1b=value1b key2b=value2b... "]]  # Pass custom parameters directly to playbooks. You may either pass parameters to all playbooks or specific ones.
  -v, [--verbose=[1|2|3]]
                                                                                                                              # Default: 0

Description:
  Plays a PowerPlay script. The entries in the script, as specified inside of a group, are run in parallel by default.

There is a short-hand ‘pp’ command you may use that has the ‘play’ task as the default. So, for example, rather than having to type:

powerplay play -p development ...

You can do instead:

pp -p development ...

In all our examples, we will use the longer ‘powerplay’ command, but you can easily substitute ‘pp’.

Example .play Script

To play around with the example .play script, Clone the Ansible Powerplay project locally:

git clone git@github.com:flajann2/ansible-powerplay.git

and go to the examples directory to find test.play.

Submitting your example .play scripts

Please feel free to do pull requests of your scripts or submit them to me as Gist snippets and I will include them if they are good.

Concurrency

We offer a finely controllable concurency model in the DSL with groups. The short of it is that a group may be marked as :sync or :async. All contents of a :sync group shall be executed serially. All contents of an :async group shall be executed concurrently.

As you can now nest groups, and that each group is either synchronous or asynchronous, how these interact requires a bit of understanding as to how the sync and async job queing mechanism in PowerPlay actually works.

The Gory Details behind how :sync and :async

Internally, we have two job queues, sync_jobs and async_jobs. We also have – at least conceptually – two run queues, sync_runs and async_runs, to reflect queues of currenly running jobs, or books. A “job” or a “book” represent an actual Ansible Playbook being run, or waiting to be run.

enqueue	deque and run ‘queues’
sync_jobs	sync_runs
async_jobs	async_runs

As well, we have the following queuing rules. Please note that “iff” is the mathematical “iff”, meaning “if and only if”.

rule	details	behavior
enqueue	async job	iff sync_jobs is empty and all sync_runs completed
	sync job	iff async_jobs is empty and all async_runs completed
dequeue and run	async queue	grab everything and run it concurrently
	sync queue	grab one at a time and run it until it completes

Note that “dequeue and run” flips back and forth between working on the sync and async queues. Never both simultaneously.

Nested Groups

You can appreicate that understanding the behavior and “interaction” of nested queues can get pretty hairy, but just keep in mind the rules above, as your nesting will rigorously adhere to the logic above, even as it descends into the queues. The group designation only directly affects its immediate jobs, or books. It does not directly affect the books in its nested children.

To ensure that the groups are themselves executed synchronously if the parent group is synchronous, internally we insert :noop book types to ensure the algorithm behaves itself accordingly. Otherwise, two consecutive async groups would appear to come from one async group.

Implemention of the Execution Planning [authoritative]

In actuality, what we do at the DSL processing level is decide whether or not a book is a sync book or async book. We generate the actual command line code at that point, and create a pair [:sync, book] or [:async, book] and push that into the planning queue, which is a FIFO queue.

(Note the the following is conceptual. In actuality, the info is all inside the book object.)

book	enqueue to FIFO planning_queue
sync group	[:sync, bash string]
async group	[:async, bash string]
naked	[:sync, bash string]

We determine what execution planning a book gets by its immediate grouping. A group’s default is :async. Naked books are :sync by default. We do this to be intuitive about how things work in the DSL. You should explicitely have to specify what’s going to be async, since that is the “more dangerous” mode.

dequeue from FIFO	action
[:sync, bash string]	join all entries in async_run_queue, clear that queue, and then execute and join bash string task
[:async, bash string]	execute and enqueue to async_run_queue

This simplifies the algorithm and makes it easier to understand, and should result in a more intuitive grasp on how to write the PowerPlay.

Scenarios

Contributing to ansible-powerplay

Your parcipitation is welcome, and I will respond to your pull requests in a timely fashion as long as I am not pulling an “Atlas” at my current job! lol

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
Fork the project.
Start a feature/bugfix branch.
Commit and push until you are happy with your contribution.
Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Copyright

The Junkyard

This area should be ignored, just a place for me to keep old snippets of code and other notes that will be of relevance to no one else.

Old execution planning model

# old code and will be deleted
playbooks do |pname, playbook|
  group_threads = []
      puts "PLAYBOOK #{pname} (group=#{Play::clopts[:group]}) -->"
      groups playbook do |group|
        tg = nil
        group_threads << (tg = Thread.new {
                            puts "    GROUP #{group.type} (book=#{bucher}, cg=#{congroups}) -->"
                            book_threads = []
                            errors = []
                            group.books.each { |book| get_book_apcmd(book, bucher, book_threads, errors) }
                            book_threads.each{ |t| t.join }
                            unless errors.empty?
                              errors.each do |yaml, cmd, txt|
                                puts '=' * 30
                                puts ('*' * 10) + ' ' + yaml
                                puts txt
                                puts '-' * 30
                                puts cmd
                              end
                              exit 10
                            end
                          })
        # Always wait here unless we're concurrent
        group_threads.join unless congroups
      end
      group_threads.each{ |t| t.join }
    end

gcaracuel/ansible-powerplay