Parallelize `create_rank_to_trace_dict`
yifanliu112 opened this issue · 0 comments
yifanliu112 commented
🚀 Motivation and context
When parsing large number of trace files, create_rank_to_trace_dict becomes the bottleneck since it needs to read every trace file sequentially to find the rank numbers.
Description
Unlike other parts in the trace parsing logic, there is no option to parallelize create_rank_to_trace_dict. Parallelize create_rank_to_trace_dict would improve the trace parsing performance.
Alternatives
No response
Additional context
No response