error saving figures after preprocessing

Question

error saving figures after preprocessing

Closed this issue 4 years ago · 11 comments

hi all, thanks for the great work!

with a recent dataset (~2hrs continuous recording, 64ch) my preprocessing pipeline (PREP + MARA) breaks when trying to save the output matlab figure.
this is annoying, because apparently all the work is done but then not stored due to the error..
.jpg (looking alright) and corrupted .fig file are present in the output folder.

i am using matlab 2018a and current dev version of automagic via the GUI.
i did not get this bug with a different much smaller dataset (2 x 10min eeg recordings from the same amplifier).

here the full command line message list:

Start preprocessing all dataset
Processing file sub-03_sub-03_ses-01_task-aeEEG_eeg ...(file 1 out of 1)
Running Prep...
Using option file in directory ~
Preliminary detrend to compute reference
pop_eegfiltnew() - performing 3301 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase)
firfilt(): |====================| 100%, ETE 00:00
Line noise removal
Starting parallel pool (parpool) using the 'local' profile ...
connected to 2 workers.
Find reference
pop_eegfiltnew() - performing 3301 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase)
firfilt(): |====================| 100%, ETE 00:00
Removing 11 channel(s)...
Interpolating missing channels...
Iteration: 1
Removing 11 channel(s)...
Interpolating missing channels...
Iteration: 2
Robust reference doneRemoving 11 channel(s)...
Interpolating missing channels...
pop_eegfiltnew() - performing 3301 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase)
firfilt(): |====================| 100%, ETE 00:00
Removing 11 channel(s)...
Interpolating missing channels...
pop_eegfiltnew() - performing 3301 point highpass filtering.
pop_eegfiltnew() - transition band width: 1 Hz
pop_eegfiltnew() - passband edge(s): 1 Hz
pop_eegfiltnew() - cutoff frequency(ies) (-6 dB): 0.5 Hz
pop_eegfiltnew() - filtering the data (zero-phase)
firfilt(): |====================| 100%, ETE 00:00
Detecting bad channels using routines of clean_raw_data()...
Perform Filtering...
Performing MARA ICA (this may take a while...)
Run ICA
MARA is computing features. Please wait
.Octave functions should not run on Matlab
Removing path ...
...........................................
Features ready
Computing projection ....
eeg_checkset: recomputing the ICA activation matrix ...
Computing rating for sub-03_ses-01_task-aeEEG_eeg

Error using save
Error closing file
/Users/dominik.welke/Work/02_projects/04_ae_eeg_exploration/3_data/eeg/eeglab_aeeeg1_nophysio_results/sub-03/sub-03_ses-01_task-aeEEG_eeg.fig.
The file may be corrupt.

Error in matlab.graphics.internal.figfile.FigFile/write (line 32)
save(obj.Path, obj.MatVersion, '-struct', 'SaveVars');

Error in savefig (line 84)
FF.write();

Error in Block/saveFiles (line 1288)
savefig(fig1, strcat(self.imageAddress, '.fig'), 'compact'); % as fig

Error in Block/preprocess (line 578)
self.saveFiles(EEG, automagic, fig1, fig2, fig3);

Error in Project/preprocessAll (line 501)
[EEG, automagic] = block.preprocess();

Error in mainGUI>runpreprocessbutton_Callback (line 809)
project.preprocessAll();

Error in gui_mainfcn (line 95)
feval(varargin{:});

Error in mainGUI (line 61)
gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)mainGUI('runpreprocessbutton_Callback',hObject,eventdata,guidata(hObject))
Error while evaluating UIControl Callback.

i didnt dig into this deeper, yet, but similar problems can arise with other file types when choosing an old format version.
might be related to memory issues

if any of you knows this bug or has an easy solution, let me know :)

Answer 1 · 2021-04-15T09:15:36.000Z

Hi Dominik

we are aware of this problem. Matlab has hard times handling large (>2Gb) figures. To overcome this problem we downsample the data for plots (only plots!) in preprocess.m (from Line 435). As the downsampling affects only plots, you can downsample the data even further to reduce size of the .fig file.
How large are your files?

Best,
Dawid

Answer 2 · 2021-04-15T11:42:12.000Z

thanks for the swift reply @ksgfan !

To overcome this problem we downsample the data for plots (only plots!) in preprocess.m (from Line 435). As the downsampling affects only plots, you can downsample the data even further to reduce size of the .fig file.

ah, great.. is this the Downsampling Rate field in the Òptions section of the GUI? i was wondering about it already, must have missed this detail in the publication..
maybe a minor GUI modification to label the box Plot Options would make it more intuitive

How large are your files?

my files are roughly 1.5 - 3 GB.
i have 1kHz sampling rate: which downsampling factor would you suggest - 4, 10? i'd like to avoid trial and error, obviously :)

best,
Dominik

Answer 3 · 2021-04-15T12:32:55.000Z

ah, great.. is this the Downsampling Rate field in the Òptions section of the GUI?

No, there you can downsample your whole data, not only for plots. The downsampling for plots is done automatically in preprocess.m, from line 435, if a file is larger than 350 or 700 Mb. I did not expect bigger files. For 3GB you need to downsample by a factor of 16-20 probably.

In order to do so, you need to go to preprocess.m, line 445 and change all '4' to '16' or '20'.

Let me know if it works.

Best,
Dawid

Answer 4 · 2021-04-15T13:05:47.000Z

hmm, but now looking into this your wiki says so (https://github.com/methlabUZH/automagic/wiki/Configurations#additional-options):

Additional options:

The downsampling only affects the visual representation of your data. A higher downsampling rate will shorten loading times and reduce memory need. In general, a downsampling rate of 2 is a good choice. But this still depends on the size of the EEG.

Answer 5 · 2021-04-15T15:24:12.000Z

you're right, sorry.

Still, if you want Automagic to work in this case, you should edit the preprocess.m as mentioned above. Changing the Options won't work.

Anyway, that is an important issue, we have to think about it and find a better solution.

Answer 6 · 2021-04-15T15:37:27.000Z

yes, i just tried it and changing the GUI parameter didnt work..

when going in the preprocessing.m code, i found another issue. currently it says:

if filesize > 350
(...)
elseif filesize > 700
(...)
end

this means that the elseif is not reached, even if filesize IS >700, because its also >300. hence its either no downsampling, or factor 2.
this needs to be reversed:

if filesize > 700
(...)
elseif filesize > 350
(...)
end

if you like i can open a quick PR to fix this, maybe adding a third elseif for files larger than 1.5 gb.
if it should take the info from the GUI its probably faster, if you do this yourself because i'm not familiar with where the info is stored

Answer 7 · 2021-04-15T15:51:28.000Z

Thanks for pointing that out! I committed a new version of preprocess.m to development branch. it should work now.

if it should take the info from the GUI its probably faster, if you do this yourself because i'm not familiar with where the info is stored

That is not so easy to do, but I am working on it.

Answer 8 · 2021-04-15T15:53:49.000Z

That is not so easy to do, but I am working on it.

i can imagine..

thanks, I'll test and let you know!

Answer 9 · 2021-04-20T12:36:08.000Z

ah sorry, i wanted to give feedback!
yes, this modification works for me (i changed it in my fork).

in your commit you used exclusive < and > though, so the code is not covering the unlikely but possible cases of filesize == 2000 (or 1500, 1000, 700):

the & filesize < ... is not needed anyway, since matlab evaluates the if/elseifs sequentially

if filesize > 2000
  (...)
elseif filesize > 1500 & filesize < 2000
  (...)
elseif filesize > 1000 & filesize < 1500
  (...)
elseif filesize > 700 & filesize < 1000
  (...)
elseif filesize > 350 & filesize < 700
  (...)
end

cheers, Dominik

Answer 10 · 2021-04-28T15:19:20.000Z

Hi Dominik

we removed the .fig file from Automagic, as it was often too large. Now we just merge 2 jpg files after interpolation of bad channels. It should save a lot of time and computational power. I updated the dev and master branch just now.

Best,
Dawid

Answer 11 · 2021-04-28T17:18:43.000Z

hi Dawid

thanks for the update. sounds reasonable

best, Dominik