Ansible Powerplay
Powerplay allows you to run multiple Ansible playbooks in parallel. Depending on how you organize your playbooks, this can be a solid win. I basically before this had been doing a playbook with multiple includes for other playbooks representing different servers in our stack. Playbook launching of playbooks is slow and very serial.
Basically, the playbooks are all contained, so no interdependencies. And in my case, running in the cloud, so no reason why they can’t be running in parallel
Powerplay allows you to specify vars common to all playbooks, and also vars specific to some playbooks so by which you can make your setup very DRY.
All the Ansible playbooks are executed in seperate processes, and thus avoiding a number of the “side effects” you would normally encounter with running multiple playbooks with Ansible includes.
For example, here is Powerplay integrated with tmux:
Release Notes
Please see Release Notes.
Hilights
DSL
The version 1.x releases adds new features to the DSL, most notably, nestable groups, and being able to label each group as :sync or :async.
STDOUT from ansible-playbook
The capture of the output from ansible-powerplay is handled a bit more intelligently. If you do not specify the –tmux (or -m) option, all output is now currently captured by Powerplay and redumped to the console.
Because you may still want to see the color from ansible-powerplay, you can alter ansible.cfg and add the following line:
- force_color = 1
Please see New STDOUT capturing with 1.0.x
Group Sequence Numers
We now allow groups to have sequence numbers, as of 1.1. Basically, if you specify a sequence, the variable you designate will be assigned a value in each of the sequence, with the group re-excuted. for example:
group :first, "async group with sequencing",
seq: { iter: [1, 5, 9, :dodo] } do
book :nat, "nat.yml"
book :dat, "dat.yml"
book :rat, "rat.yml"
end
as you can see (in the development.play sample) the variable “iter” will be successively assigned the element in the [] array, with the underlying playbooks called. The is the function equivalent of the following:
group :first, "async group without sequencing" do
configuration do
iter 1
end
book :nat, "nat.yml"
book :dat, "dat.yml"
book :rat, "rat.yml"
end
group :first, "async group without sequencing" do
configuration do
iter 5
end
book :nat, "nat.yml"
book :dat, "dat.yml"
book :rat, "rat.yml"
end
group :first, "async group without sequencing" do
configuration do
iter 9
end
book :nat, "nat.yml"
book :dat, "dat.yml"
book :rat, "rat.yml"
end
group :first, "async group without sequencing" do
configuration do
iter :dodo
end
book :nat, "nat.yml"
book :dat, "dat.yml"
book :rat, "rat.yml"
end
As you can see, the new sequencing can be quite succinct.
Features and Cavets
Integration with TMUX
When running multiple Ansible Playbooks concurrently, one would like to be able to see the output of each in a reasonable manner. To faciliate this in this initial realse, we shall make heavy use of TMUX panes to dump the output.
So basically, you need as many panes as you have concurrent Ansible Playbooks in this initial release. In subsequent releases, Curses will be directly leveraged to create “tabs” for the multiple output streams. We may even do this, still, through TMUX.
Your input on this is strongly encouarged. We will not be supporting Screen at all. Sorry.
New STDOUT capturing with 1.0.x
The new capture, while properly capturing the STDOUT of concurrent async runs, does not display until ansible-playbook completes. If you are like me, you’ll like to see the progress as it runs. You still can using tmux with the –tmux (-m) option.
DSL Terminology & Documentation
Note that this is the DSL for version 1.x of PowerPlay. For 0.x, please see those tags in GitHub.
DSL
The DSL is straightforward as possible, simple and elegant to allow you to write your Powerplays in a DRY manner.
For examples, please see the following:
stack.play | This is loaded by default, and you must be in your current directory |
development.play | This is a fullblown Power Playbook for a hypothetical development stack. |
production.play | This is a fullblown Power Playbook for a hypothetical production stack. |
playbooks | Sample Ansible playbooks called by Powerplay. |
To run the powerplay example:
- Install Ansible Powerplay
- gem install ansible-powerplay
- Clone this project locally, then cd into the examples directory
- git clone https://github.com/flajann2/ansible-powerplay.git
- cd ansible-powerplay/examples
- source ansible-paths and run Powerplay
- source ansible-paths.sh
- powerplay play -p development -v2
Note that I deliberately left a missing “elasticsearch.yml” so you can see how Powerplay handles the errors.
configuration
You can intersperse configuration blocks anywhere, and the expected nested scoping will take effect.
playbooks
playbooks are a collection of groups, and a group defaults to async mode for its members.
Group are normally executed serially. This will allow you to organize your plays in an intelligent manner to deploy and manage resources and assets that may have to be done in a serial manner.
group
A group is a collection of books or other groups that all execute in parallel by default. Books are required to be independent of each other. If they are not, you can set them up to execute serially.
book
A book has a direct correspondence to an Ansible playbook, and will execute that Yaml file given the configuration variables as parameters.
Here is where var inheritance becomes useful. Note that all the configuration variables set at the time the book is called are all passed in as –extra-vars to Ansible Playbook. The Playbook may not need all the vars passed in, but care must be taken that no vars are used in a different manner than expected. We currently have no way of knowing which vars are needed or not, and to specifiy that would make the syntax messy and loose some of the advantages of var inheritance.
Installation
Easy installation. From command-line:
gem install ansible-powerplay
Or from a gemfile:
gem 'ansible-powerplay'
Use
Basically, cd to the root of your Ansible directory, and a .play file (see the example at: stack.play.)
You can place a config clause either globally, inside of playbooks, inside of groups, and the variable set up this way are inherited to the inner clauses, thus allowing you to keep your specifications DRYer.
For example:
# This is a global system configuration
configuration :system do
playbook_directory "playbooks"
end
Note that ‘playbook_directory’ is special, as it allows you to define the directory all of your Ansible playbooks can be found. You can also specify this anywhere you can use the configuration clause, so you may set up different playbook directories for different playbook collections.
# sṕecific configuration for :development
configuration do
stack :development
krell_type "t2.small"
servers 1
rolling 3
krell_disk_size 20
end
The above shows Ansible variables for my specialiezd setup that is geared with work with AWS. You are free to specify any variables here, which will be injected into ansible-playbook through the ‘–extra-vars’ parameter.
Here is a group clause with a single book in it:
# Groups are executed serially.
group :first, "our very first group" do
# Books within a group are executed in parallel,
# and therefore must be independent of each other.
book :nat, "nat.yml"
end
Which issues the following command to Ansible (based on the earlier configuration):
ansible-playbook playbooks/nat.yml \
--extra-vars "playbook_directory=playbooks stack=development krell_type=t2.small servers=1 rolling=3 krell_disk_size=20"
And if our group had more book entries, as in the second example:
group :second, "our second group" do
book :rabbit, "rabbitmq_cluster.yml" do
krell_type "t2.medium"
end
book :es_cluster, "elasticsearch_cluster.yml" do
esver "1.7.4"
cluster_name :es
servers 3
heapsize "2g"
krell_type "t2.medium"
krell_disk_size 200
end
end
Both the :rabbit and :es_cluster books would be executed in parallel.
Dividing up your specs in other PowerPlay files
Ruby, the underlying language, give you a lot of things for “free”, like allowing you to load other powerplay files, for example:
load 'production.play'
We mention this here for those who may not be familiar with Ruby, but may wish to section off your specifications thusly.
You don’t really need to know any Ruby, but it could increase the span of what you might want to do. To get a quick taste, please checkout Ruby in 20 Minutes.
It is also possible to leverage Ruby’s metaprogramming techniques to create templates for your specificaitons, but at some point, as time allows, I may directly support this in the DSL. Please let your wishes be known to me for this and any other feature you might want to see.
Running Powerplay
If you type ‘powerplay’ without parameters, you are greeted with:
Commands:
powerplay help [COMMAND] # Describe available commands or one specific command
powerplay play <script> -p, --play=[NAME|all] Which playbook shelf # Run the powerplay script.
powerplay ttys # list all the TMUX ptys on the current window.
Options:
-v, [--verbose=[1|2|3]]
# Default: 0
Please use the help feature to explain the subcommands and options. We shall be adding many more subcommands and options as our needs demands. If you like to see something here, please submit it as an issue on Github.
And for an example of play help, (note that this may not be up-to-date, so please run ‘powerplay help play’ on your installe version!)
powerplay help play
Usage:
powerplay play [script] -p, --play, --power, --play=[NAME[ NAME2...]|all]
Options:
-m, [--tmux=[WINDOWNUMBERopt]] # Send output to all tmux panes in the current window, or the numeric window specified.
-p, --play, --power, --play=[NAME[ NAME2...]|all] # Which PowerPlay playbooks (as opposed to Ansible playbooks) to specifically execute.
-g, [--group=[NAME[ NAME2...]|all]] # Which groups to execute.
# Default: [:all]
-c, [--congroups], [--no-congroups] # Run the groups themselves concurrently
-b, [--book=[NAME[ NAME2...]|all]] # Which books to execute.
# Default: [:all]
-u, [--dryrun], [--no-dryrun] # Dry run, do not actually execute.
-x, --extra-vars, [--extra=<BOOKNAME|all>:"key1a=value1a key2a=value2a... " [BOOKNAME2:"key1b=value1b key2b=value2b... "]] # Pass custom parameters directly to playbooks. You may either pass parameters to all playbooks or specific ones.
-v, [--verbose=[1|2|3]]
# Default: 0
Description:
Plays a PowerPlay script. The entries in the script, as specified inside of a group, are run in parallel by default.
There is a short-hand ‘pp’ command you may use that has the ‘play’ task as the default. So, for example, rather than having to type:
powerplay play -p development ...
You can do instead:
pp -p development ...
In all our examples, we will use the longer ‘powerplay’ command, but you can easily substitute ‘pp’.
Example .play Script
To play around with the example .play script, Clone the Ansible Powerplay project locally:
git clone git@github.com:flajann2/ansible-powerplay.git
and go to the examples directory to find test.play.
Submitting your example .play scripts
Please feel free to do pull requests of your scripts or submit them to me as Gist snippets and I will include them if they are good.
Concurrency
We offer a finely controllable concurency model in the DSL with groups. The short of it is that a group may be marked as :sync or :async. All contents of a :sync group shall be executed serially. All contents of an :async group shall be executed concurrently.
As you can now nest groups, and that each group is either synchronous or asynchronous, how these interact requires a bit of understanding as to how the sync and async job queing mechanism in PowerPlay actually works.
The Gory Details behind how :sync and :async
Internally, we have two job queues, sync_jobs and async_jobs. We also have – at least conceptually – two run queues, sync_runs and async_runs, to reflect queues of currenly running jobs, or books. A “job” or a “book” represent an actual Ansible Playbook being run, or waiting to be run.
enqueue | deque and run ‘queues’ |
---|---|
sync_jobs | sync_runs |
async_jobs | async_runs |
As well, we have the following queuing rules. Please note that “iff” is the mathematical “iff”, meaning “if and only if”.
rule | details | behavior |
---|---|---|
enqueue | async job | iff sync_jobs is empty and all sync_runs completed |
sync job | iff async_jobs is empty and all async_runs completed | |
dequeue and run | async queue | grab everything and run it concurrently |
sync queue | grab one at a time and run it until it completes |
Note that “dequeue and run” flips back and forth between working on the sync and async queues. Never both simultaneously.
Nested Groups
You can appreicate that understanding the behavior and “interaction” of nested queues can get pretty hairy, but just keep in mind the rules above, as your nesting will rigorously adhere to the logic above, even as it descends into the queues. The group designation only directly affects its immediate jobs, or books. It does not directly affect the books in its nested children.
To ensure that the groups are themselves executed synchronously if the parent group is synchronous, internally we insert :noop book types to ensure the algorithm behaves itself accordingly. Otherwise, two consecutive async groups would appear to come from one async group.
Implemention of the Execution Planning [authoritative]
In actuality, what we do at the DSL processing level is decide whether or not a book is a sync book or async book. We generate the actual command line code at that point, and create a pair [:sync, book] or [:async, book] and push that into the planning queue, which is a FIFO queue.
(Note the the following is conceptual. In actuality, the info is all inside the book object.)
book | enqueue to FIFO planning_queue |
---|---|
sync group | [:sync, bash string] |
async group | [:async, bash string] |
naked | [:sync, bash string] |
We determine what execution planning a book gets by its immediate grouping. A group’s default is :async. Naked books are :sync by default. We do this to be intuitive about how things work in the DSL. You should explicitely have to specify what’s going to be async, since that is the “more dangerous” mode.
dequeue from FIFO | action |
---|---|
[:sync, bash string] | join all entries in async_run_queue, clear that queue, and then execute and join bash string task |
[:async, bash string] | execute and enqueue to async_run_queue |
This simplifies the algorithm and makes it easier to understand, and should result in a more intuitive grasp on how to write the PowerPlay.
Scenarios
Contributing to ansible-powerplay
Your parcipitation is welcome, and I will respond to your pull requests in a timely fashion as long as I am not pulling an “Atlas” at my current job! lol
- Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
- Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
- Fork the project.
- Start a feature/bugfix branch.
- Commit and push until you are happy with your contribution.
- Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
- Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
Copyright
Copyright (c) 2016 Fred Mitchell. See LICENSE.txt for further details.
The Junkyard
This area should be ignored, just a place for me to keep old snippets of code and other notes that will be of relevance to no one else.
Old execution planning model
# old code and will be deleted
playbooks do |pname, playbook|
group_threads = []
puts "PLAYBOOK #{pname} (group=#{Play::clopts[:group]}) -->"
groups playbook do |group|
tg = nil
group_threads << (tg = Thread.new {
puts " GROUP #{group.type} (book=#{bucher}, cg=#{congroups}) -->"
book_threads = []
errors = []
group.books.each { |book| get_book_apcmd(book, bucher, book_threads, errors) }
book_threads.each{ |t| t.join }
unless errors.empty?
errors.each do |yaml, cmd, txt|
puts '=' * 30
puts ('*' * 10) + ' ' + yaml
puts txt
puts '-' * 30
puts cmd
end
exit 10
end
})
# Always wait here unless we're concurrent
group_threads.join unless congroups
end
group_threads.each{ |t| t.join }
end