FlatLady

Rails flat file importation process automation. Flat lady automates the following processes :

1. Map the columns of any given flat file to a denormalized schema understood by your system
2. Validate the integrity of the imported data
3. Run the importation process in the background when asked by the user

it ain’t over until the flat_lady signs

Dependencies

to automate the importation process , flat_lady uses different gems that will need to be included

In config/environment.rb:

config.gem ‘paperclip’
config.gem ‘state_machine’
config.gem ‘ar-extensions’
config.gem ‘fastercsv’
config.gem ‘haml’,
:version => ‘~> 2.0.4’

plugins :

./script/plugin install git://github.com/giraffesoft/resource_controller.git

Installation

now that the dependencies are installed, install the plugin:

ruby script/plugin install git@svn.codegenome.com:flat_lady.git

Make sure the development database exists and run the generator:

./script/generate flat_lady

A number of files will be created and instructions will be printed.

You may already have some of these files. Don’t worry. You’ll be asked if you want to overwrite them.

Run the migration:

rake db:migrate

Background worker installation

Flat Lady uses the delayed_jobs plugin to run the importation related jobs in the background. So if it’s not installed already, you’ll have to install the plugin :

./script/plugin install git://github.com/tobi/delayed_job.git

Make sure the development database exists and run the generator:

./script/generate flat_lady_background

for more information on the delayed_job usage, please refer to : http://github.com/tobi/delayed_job/tree/master.

Usage

Flat lady enables you to provide an importation interface for any model of your system.

The first step is to define a denormalized schema which will be used as a buffer to validate the data before the importation.

script/generate import_scaffold model_name field1:string field2:string ….

the second step is to define the “check_integrity” and “push!” methods of the generated model

Example :

let’s say you want to import contacts from many different csv files, but, in your app, contacts are define as follow :

class Contact < ActiveRecord::Base has_many :phone_numbers

you would start by defining a denormalized (flat) schema for contacts with many phone numbers :

script/generate import_scaffold contact name:string phone1:string phone2:string phone3:string phone4:string phone5:string

this will create the scaffold with the required migrations and routes for the ImportedContact model.

Make sure you define the 2 following methods on the ImportedContact class

def check_integrity

# this method will be called on each imported_contact directly after translating the csv file. flat_lady adds 2 integrity flags by
#default to the ImportedContact model. Those flags are : invalid_import and duplicated_import. Those flag should be set in the
# check_integrity method.

def push! #this method will be called to import the data. This is were you specify how to denormalize and import your imported_contacts

start the background worker

rake jobs:work

The users of your application should now be able to import contacts via /flat_lady/import_files/new

Cucumber Features

As your app evolves, you want to know that authentication still works. FlatLady’s opinion is that you should test its integration with your app using Cucumber.

In config/environments/test.rb:

config.gem ‘webrat’, :version => ‘= 0.4.4’ config.gem ‘cucumber’, :version => ‘= 0.3.0’ config.gem ‘thoughtbot-factory_girl’, :lib => ‘factory_girl’, :source => “http://gems.github.com”, :version => ‘1.2.1’

Vendor the gems:

rake gems:install RAILS_ENV=test rake gems:unpack RAILS_ENV=test

Don’t vendor nokogiri (due to its native extensions):

rm -rf vendor/gems/nokogiri-1.2.3

Run the Cucumber generator (if you haven’t already) and FlatLady’s feature generator:

script/generate cucumber script/generate flat_lady_features

Authors

David Fugère, Benjamin Thouret.

Questions?

Ask the mailing list_

dfugere/flat_lady