EDIot (Electronic Data Interchange oriented transformer)

This gem provides the ability to transform an EDI X12 ANSI 834 formatted file (row based) to flattened CSV format (column based). You can find details on this format here and here.

Installation

Add this line to your application's Gemfile:

gem 'grnds-ediot', git: 'https://github.com/ConsultingMD/ediot.git', tag: '<sha-here>'

And then execute:

$ bundle

Or install it yourself as:

$ gem install grnds-ediot

Usage

Demo

See rake tasks for examples of processing files in bulk or streaming:

$ rake -T

Rake tasks expect a tmp directory to be present in the project directory (e.g. tmp/). Some sample files can be found in the spec/support directory.

$ rake demo:with_stream
$ rake demo:with_file

IRL

Here's an example usage to create a CSV from an 834 file. The parser is designed for streaming data. Feed it an IOStream compatible object.

require 'csv'
require 'grnds/ediot'

file_enum = Grnds::Ediot::Parser.lazy_file_stream('tmp/my_834_file.txt')
File.open('tmp/my_csv_file.csv', 'w') do |out_file|

  # create a parser with the default 834 definition
  parser = Grnds::Ediot::Parser.new

  # record definition contains keys for each record row
  column_keys = parser.row_keys

  # write the csv header row first
  out_file << CSV::Row.new(column_keys, column_keys, true)

  # parser takes a fileIO object and will read the file
  # lines until EOF. As the reading cursor advances for
  # each complete record it finds it will yield an array
  # representing the row object.
  parser.parse(file_enum) do |row|
    out_file << CSV::Row.new(column_keys, row)
  end
end

834 file definition

In order to properly parse an 834 file we have a simple definition object. Abstracting the structure of the 834 file in this way makes it easy to change it also makes the library code easy to test (see spec files for examples).

For context, an example 834 record looks like this:

INS*Y*18*030*AB*A***FT**N*******0
REF*0F*00000000
REF*23*800188350
REF*ZZ*00000000W
DTP*356*D8*20020128
DTP*336*D8*20040126
NM1*IL*1*SMITH*JOHN*Q***34*000000000
PER*IP**HP*5785552630*EM*JOHNBOY@CRAZY8.NET*CP*5735552630
N3*387 EAST WEST ROAD
N4*LONELY CREEK*M0*68786
DMG*D8*19500803*M
HLH*N*0*0
HD*030**HLT*        0920200300000000000000000000000000000000  *ESP
DTP*348*D8*20160101
AMT*D2*6000
REF*1L*WY 00222D 0001419020 2100572 N65533    WMO

Note: This gem was updated to only support ~ based line separators in files (i.e no newlines in the file, only tildes). The file faker and all the specs have been updated to reflect this change. If you want to use the library for a file that does not have ~ based line endings you can pass this in to the lazy_file_stream class method.

For example:

file_enum = Grnds::Ediot::Parser.lazy_file_stream(INPUT_FILE_PATH, "\n")

Based on this example record, the out-of-the-box definition is:

DEFINITION = {
  INS: { size: 18 },
  REF: { occurs: 5, size: 3 },
  DTP: { occurs: 3, size: 4 },
  NM1: { occurs: 2, size: 10 },
  PER: { size: 9 },
  N3:  { size: 2 },
  N4:  { size: 4 },
  DMG: { size: 4 },
  HLH: { size: 4 },
  HD:  { size: 6 },
  AMT: { size: 3 }
}

The definition hash has a few key features.

Each entry in the definition hash represents a row type we want to parse. If the row key is not the hash, it won't end up in your output.
The first entry in the definition is the header row. The parser will scan the file lines until it reaches one of these. Since the format type of this 834 file is "unbounded" (see reference), we don't know the current record has ended until we see the start of the next record.
Each entry has a key (e.g. INS). This corresponds to the first key in the 834 "segment." The hash key must be a symbol, be in all caps, and it must have a hash as the value. That hash must contain the key :size and have an integer value that represents the number of "elements" in the segment row (each "element"is separated by the '*' character). Optionally, you can pass in a second key :occurs this is the MAX number of times you expect to see this segment type in one record. If during record parsing this number is exceeded it will throw an error. Likewise for going over the number of elements designated by :size. LESS IS OKAY. Just not more. More elements than expected will throw off the pivoting from rows to columns. It would be bad. Bad like mass-hysteria, dogs and cats living together, etc...

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Testing

Uses RSpec. If you find bugs or make changes, write tests first. To run the test suite:

$ rake spec

How to Create a Release

Releases happen in CircleCI when a tag is pushed to the repository. To create a release, you will need to do the following:

Change the version in lib/grnds/ediot/version.rb to the new version and create a PR with the change.
Once the PR is merged, switch to the master branch and git pull.
git tag <version from version.rb>
git push origin --tags CircleCI will see the tag push, build, and release a new version of the library.

runongirlrunon/ediot