Implement memory management in shuffle writer
andygrove opened this issue · 1 comments
andygrove commented
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There is no memory management in shuffle writer
Describe the solution you'd like
Blaze implements file-based shuffle with memory management in https://github.com/blaze-init/blaze/blob/master/native-engine/datafusion-ext/src/shuffle_writer_exec.rs
Perhaps we can learn from that?
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
See discussion in apache/datafusion#3493 (comment)
mingmwang commented
I will take care of this ticket.