Implement an adapter to allow using a source as a transform
Closed this issue · 1 comments
WIth the publication of Kiba v2 and the introduction of Kiba::StreamingRunner
, we can replace the following types of CSV sources:
class CSVSource
def initialize(dir_pattern:, csv_options:)
def each
Dir.glob(dir_pattern).each do |file|
CSV.foreach(file, csv_options) do |row|
yield(row.to_h)
end
end
end
with these more decoupled & composable ones:
source Kiba::Common::Sources::Enumerable, -> { Dir["input/*.csv"] }
transform SomeCSVParsingTransform, csv_options: options
But this means that if you have an existing CSV source (responding to each
), you cannot use it directly as a CSV transform (which must respond to process(row)
).
It is clear that the role of sources & transforms is gradually becoming much more similar with v2.
I want to be able to write something like:
transform SourceTransformAdapter, SomeCSVSource, csv_options: options
We could even imagine that once some option is opted-in, one could just write:
transform SomeCSVSource, csv_options: options
# or maybe
transform source: SomeCSVSource, csv_options: options
and then the parser would transparently detect if the instance respond_to? each
or process
to automatically apply the expected behaviour.
All in all I need some time to make some tests on real-life apps and see what is the best way to implement this, but this would be a net win to write more reusable components.