evanmiller/fmptools

Running extremely slow on large fmp12 data file

ye opened this issue · 5 comments

ye commented

Hi,

I've downloaded the source code from the git today and compiled it on a Mac Min (Apple M1 chip) with 8 GB physical memory.

The data file is a 9.3GB .fmp12 data file. I have been running it for over a couple of hours now (still running). Is there anyway to speed up the performance?

Thanks.
Screenshot 2022-12-12 at 7 38 30 PM

Screenshot 2022-12-12 at 7 38 56 PM

evanmiller commented

There's an option in libxlsxwriter to write directly to disk instead of holding the spreadsheet in memory. I'm guessing it will help here. The option is described in https://libxlsxwriter.github.io/working_with_memory.html

  • 👍1
evanmiller commented

You also might try converting to a SQLite file to see if that's successful.

  • 👍1
ye commented

Sounds good. I will report back when I give those two a try.

evanmiller commented

FYI I have made the constant_memory change to the master branch. Try pulling the latest code and see if it runs any better.

evanmiller commented

Please try the latest release and open a new issue if you continue to experience slowness.