About Iron Fixture Extractor

For extracting complex data from staging and production databases to be used for automated testing.

Its best when:

your data is too complex for factories
creating and maintaining manual fixtures is cumbersome and brittle

Use cases

Pulling data from a staging database containing vetted data that has been built up by the development team, users, or business analysts to be loaded and used as "archetypical" data structures in test cases or demos.
Taking snapshots of production data that has triggered app exceptions to be more closely inspected and incorporated into test cases.

How it works

Feed it an array of ActiveRecord objects or ActiveRelation object and it will allow you to:

extract data to .yml fixtures
load it into a database or memory
rebuild .yml fixtures from a saved ActiveRelation extraction query.

Usage

Extract fixture set (typically run in a irb console)

Fe.extract 'Post.includes(:comments, :author).limit(1)', :name =>  'first_post_w_comments_and_authors'
# or for multi-model extraction something like this:
x = '[UserRole.all, Project.includes(:contributors => [:bio])]'
Fe.extract(x,:name => :all_permissions_and_all_projects)

Load fixture set into database (typically run in a "setup" test method )

Fe.load_db(:first_post_w_comments_and_authors)

If your fixture set is huge, you can avoid loading particular tables with:

Fe.load_db(:first_post_w_comments_and_authors, :only => 'posts')

Fe.load_db(:first_post_w_comments_and_authors, :except => ['comments'])

You can also load to a table name different than the source they were extracted from via a Hash or Proc:

Via Proc: (this will add "a_prefix_" to all target tables)

Fe.load_db(:first_post_w_comments_and_authors, :map => -> table_name { "a_prefix_#{table_name}" })

Via Hash: (just maps posts to different table, the others stay the same)

Fe.load_db(:first_post_w_comments_and_authors, :map => {'posts' => 'different_posts'})

Load particular fixture into memory (typically used to instantiate an object or build a factory)

# 'r1' is the fixture's name, all fixture names start with 'r', 1 is the id
Fe.get_hash(:first_post_w_comments_and_authors, Post, 'r1')

# You can specify :first, or :last to the last arg
Fe.get_hash(:first_post_w_comments_and_authors, Comment, :first)

# Get the hash representation of the whole fixture file
Fe.get_hash(:first_post_w_comments_and_authors, Comment, :all)

# Get an array of hashes stored in a fixture file
Fe.get_hashes(:first_post_w_comments_and_authors, Comment)

This feature is used to instantiate objects from the hash or define factories like:

# Create factory from a particular hash within a fixture file
Factory.create(:the_post) do
  h=Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
  name h.name
end

or

# Create an instance
h=Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
ye_old_post=Post.new(h)

Rebuild fixture files associated with the initial extraction (also doable via rake task in Rails)

Fe.rebuild(:first_post_w_comments_and_authors)
# Make sure to `diff` your test/fe_fixtures dir to see what has changed in .yml files

Truncate tables associated with a fixture set (if you're not using DatabaseCleaner)

Fe.truncate_tables_for(:first_post_w_comments_and_authors)

Installation

Add this line to your application's Gemfile:

gem 'iron_fixture_extractor'

And then execute:

$ bundle

Or install it yourself as:

$ gem install iron_fixture_extractor

Advanced Usage/Changing fe_manifest.yml for fixture set

Each extracted fixture set has a fe_manifest.yml file that contains details about:
- The ActiveRelation/ActiveRecord query to used to instantiate objects to be serialized to .yml fixtures.
- The models, table names, and row counts of records in the fixture set

By modifying the :extract_code: field, you can change the extraction behavior associated with .rebuild. It can be handy if you want to add data to a fixture set.

Dirt Simple Shiznit

The essense of the Fe.extract "algorithm" is:

for each record given to Fe.extract
  recursively resolve any association pre-loaded in the .association_cache [ActiveRecord] method
  add it to a set of records keyed by model name
write each set of records as a <TheModel.table_name>.yml fixture
write a fe_manifest.yml containing original query, row counts, etc

Typical Workflow

Data extracted from a dev, staging, or production db is needed
Open rails console in the appropriate environment
Monkey with ActiveRecord queries to collect the data set you want to use in your test case.
Represent the ActiveRecord query code as a string, i.e. x='[User.all,Project.includes(:author).find(22)]'
Extract the data into fixtures, Fe.extract(x,:name => :some_fixture_set_name)
Open up test/fe_fixtures/some_fixture_set_name and poke around the yml files to make sure you've captured what you need. Tweak extract_name if you need to and .rebuild
In your test case's setup method:

Fe.load_db(:some_fixture_set_name) ...then load a instance var to test against: ...in this case 22 is the id of a fixture that has just been loaded @the_project = Project.find(22) or Fe.execute_extract_code(:some_fixture_set_name).first
In your test case's teardown method:

DatabaseCleaner.clean... or Fe.truncate_tables_for(:some_fixture_set_name)
In your test case require 'debugger'; debugger; puts 'x'...inspect @the_project or whatever does the loaded object and db state have the fixtures you want.
Once things seem to be working-ish in your tests, add the fixtures to source control + test case that uses them.

Gem Compatibility

Works on MRI 1.9.3 and 1.9.2
Does not work on JRuby, 1.8.7

Contributing

In a nutshell:

git clone     # get the code
cd <the dir>
rake          # run the tests
# make a spec file and hack.

See spec/README_FOR_DEVELOPERS.md for more details.

TODO: JOE REPLACE THE ABOVE CONTENT WITH THIS METHOD

Alternatively, another way to lower the barrier to contributing is to submodule the Gem into your project and hack in the features you need to support your app specific, then add a test case to the Gem itself that illustrates your change...

TODO: Write a "For Rspec users", include an initializer:

# config/initializers/iron_fixture_extractor.rb
Fe.fixtures_root = 'spec/fe_fixtures' if defined?(Fe)

TODO: Write something up about bug with has_many :through

If you have a query that utilizes a has_many :through. Make sure to put the table that facilitates the :through AFTER the one that uses it. ie.

NOT WORKING

query='Etl::Receipt.includes(:audits, :instcd_to_jobid_mappings, :dars_job_queue_lists => {:job_queue_runs => [:job_queue_outs, {:job_queue_reqs => {:job_queue_subreqs => :job_queue_courses}}]})' t=Fe.extract(query,:name => 'poo')

WORKING

query='Etl::Receipt.includes(:audits, {:dars_job_queue_lists => {:job_queue_runs => [:job_queue_outs, {:job_queue_reqs => {:job_queue_subreqs => :job_queue_courses}}]}}, :instcd_to_jobid_mappings)' t=Fe.extract(query,:name => 'poo')
Beers, kudos, praise, and glory for a developer who can find the reason for this and a fix...I tried, but couldn't figure it out, hence the work around.

Footnotes

I used various ideas from the following blog posts, gists, and existing ruby gems, thanks to the authors of these pages:

Author

Joe Goggins

habanerohq/iron_fixture_extractor