apache/datafusion-ballista

Implement memory management in shuffle writer

andygrove opened this issue · 1 comments

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There is no memory management in shuffle writer

Describe the solution you'd like

Blaze implements file-based shuffle with memory management in https://github.com/blaze-init/blaze/blob/master/native-engine/datafusion-ext/src/shuffle_writer_exec.rs

Perhaps we can learn from that?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
See discussion in apache/datafusion#3493 (comment)

I will take care of this ticket.