Make it easier to rely on the new StreamingRunner
Closed this issue · 1 comments
thbar commented
Just food for thought at this stage.
In #15, I've added tests to showcase how one can use the SourceTransformAdapter
& Kiba v2 StreamingRunner
to provide "multi input" support for any source without "multi" support baked in, e.g:
source Kiba::Common::Sources::Enumerable, -> { Dir[File.join(dir, '*.csv')] }
transform do |r|
[
Kiba::Common::Sources::CSV,
filename: r,
csv_options: { headers: true, header_converters: :symbol }
]
end
transform Kiba::Common::Transforms::SourceTransformAdapter
This could be shortened a bit though, with something like:
source Kiba::Common::Sources::Enumerable, -> { Dir[File.join(dir, '*.csv')] }
transform Kiba::Common::Transforms::SourceTransformAdapter, config: -> do |r|
[
Kiba::Common::Sources::CSV,
filename: r,
csv_options: { headers: true, header_converters: :symbol }
]
end
Or even with a bit of DSLExtensions
sugar:
enumerable_source -> { Dir[File.join(dir, '*.csv')] }
multi_source Kiba::Common::Sources::CSV, -> { |f| filename: f, csv_options: xxx }
thbar commented
After some time using the StreamingRunner in production, I don't feel this is strictly necessary at this point (especially given that anyone can implement such extensions on a case-by-case basis). I will close for now & will revisit if the need feels more acute.