thbar/kiba-common

Implement an adapter to allow using a source as a transform

Closed this issue · 1 comments

thbar commented

WIth the publication of Kiba v2 and the introduction of Kiba::StreamingRunner, we can replace the following types of CSV sources:

class CSVSource
  def initialize(dir_pattern:, csv_options:)

  def each
    Dir.glob(dir_pattern).each do |file|
      CSV.foreach(file, csv_options) do |row|
        yield(row.to_h)
      end
  end
end

with these more decoupled & composable ones:

source Kiba::Common::Sources::Enumerable, -> { Dir["input/*.csv"] }

transform SomeCSVParsingTransform, csv_options: options

But this means that if you have an existing CSV source (responding to each), you cannot use it directly as a CSV transform (which must respond to process(row)).

It is clear that the role of sources & transforms is gradually becoming much more similar with v2.

I want to be able to write something like:

transform SourceTransformAdapter, SomeCSVSource, csv_options: options

We could even imagine that once some option is opted-in, one could just write:

transform SomeCSVSource, csv_options: options
# or maybe
transform source: SomeCSVSource, csv_options: options

and then the parser would transparently detect if the instance respond_to? each or process to automatically apply the expected behaviour.

All in all I need some time to make some tests on real-life apps and see what is the best way to implement this, but this would be a net win to write more reusable components.

thbar commented

Implemented in #10.