Can we support custom split logic for `RollingWriter`
Closed this issue · 3 comments
nooberfsh commented
RollingWriter::should_split
split arrow writer based some configs which have some limitation, e.g. when user want split the arrow writer after the current row num >= 1000.
Can we provide an API like this:
pub WriteStats {
current_row_num: usize,
current_written_size: usize,
}
impl RollingWriterBuilder {
fn with_split(mut self, split_f: impl Fn(&WriteStats) -> bool) -> Self {
...
}
}
ZENOTME commented
Actually the user can custom their own file writer. But I think that it's a good suggestion. How do you think? @liurenjie1024
liurenjie1024 commented
It's ok to me to have a pluggable split policy.