TGSAI/mdio-python

Reduce memory requirements for segy cutting

Closed this issue · 2 comments

mdio_to_segy requires a significant amount of memory for large datasets. This seems related to dask/dask#10535

Test dask release: 2023.10.0

Is it possible to have a minimal reproducer for this memory increase? Note that we just merged another change to the relevant code sections, see dask/dask#10660 which I hope may be more stable. I would still be interested in a reproducer for your problem

Edit: I misread the issue. I thought #10535 was the cause for a regression in your code base but it appears to be the opposite. Still, I recommend checking out the new version as well. I believe this should be released this Friday. Please ping me if it doesn't work for you

@fjetter , thanks for the interest. I'll close this issue as bumping to dask >=2023.10.0 has resolved this.