thbar/kiba-common

Make it easier to rely on the new StreamingRunner

Closed this issue · 1 comments

thbar commented

Just food for thought at this stage.

In #15, I've added tests to showcase how one can use the SourceTransformAdapter & Kiba v2 StreamingRunner to provide "multi input" support for any source without "multi" support baked in, e.g:

source Kiba::Common::Sources::Enumerable, -> { Dir[File.join(dir, '*.csv')] }

transform do |r|
  [
    Kiba::Common::Sources::CSV,
    filename: r,
    csv_options: { headers: true, header_converters: :symbol }
  ]
end

transform Kiba::Common::Transforms::SourceTransformAdapter

This could be shortened a bit though, with something like:

source Kiba::Common::Sources::Enumerable, -> { Dir[File.join(dir, '*.csv')] }

transform Kiba::Common::Transforms::SourceTransformAdapter, config: -> do |r|
  [
    Kiba::Common::Sources::CSV,
    filename: r,
    csv_options: { headers: true, header_converters: :symbol }
  ]
end

Or even with a bit of DSLExtensions sugar:

enumerable_source -> { Dir[File.join(dir, '*.csv')] }

multi_source Kiba::Common::Sources::CSV, -> { |f| filename: f, csv_options: xxx }
thbar commented

After some time using the StreamingRunner in production, I don't feel this is strictly necessary at this point (especially given that anyone can implement such extensions on a case-by-case basis). I will close for now & will revisit if the need feels more acute.