testing failure due to ram shortage

[epona7471]

Hello, I am trying this repo and had few blocking points.

I am operating this repo at Colab Pro, google drive environment (RAM is around 25G)

In file 'Training, Validation, Testing.ipynb', at Testing section, I am constantly having RAM shortage runtime error.

I only used a part of MAESTRO(2017, 2015) data for training, and it is gonna worse if I use a full dataset,

because RAM shortage runtime error also occurred at velocity model training.

Can you tell me what was your operating environment?

Is there any way to reduce the burden of memory while training and testing without down-sizing dataset?

Thank you for your interest. It would be much appreciated if somebody can help me out with this.

[BShakhovsky]

Hi,

My laptop parameters are:
RAM: 16Gb
Processor: Intel Core i7-7700HQ CPU 2.80GHz
GPU: NVidia GeForce GTX 1050 Ti
  Dedicated GPU memory 4Gb
  Shared GPU memory 8Gb

I have not really deeply looked at my templates for a couple of years. But as I remember, I also had memory errors during testing. That is why I calculate precision/recall/F1-score for each single MAESTRO musical piece, and then print average values for each year (2004, 2006, etc.). And on the next cell with ROC-curve graphs, I make calculations only for 2014 part of MAESTRO.
Performing testing on the full dataset at once is impossible on my laptop with 16Gb RAM. Not sure about 25Gb in your case, but most probably it will not be enough either.

During testing, are you also getting ResourceExhaustedError, or something like the following:

MemoryError: Unable to allocate 15.1 GiB for an array with shape (28346, 626, 229) and data type float32

If yes, then reducing the batch size may help. For example, during testing, there is a batch size parameter in "model.predict". It is 32 in the following lines of code:

onProb, offProb, volProb = map(lambda mod: mod.predict(m, 32, 1), [onsetsModel, offsetsModel, volumesModel])
actProb = activesModel.predict([m, onProb, offProb], 32, 1)

And it is 16 in the next cell in the following block of code:

onProb2014, offProb2014, volProb2014 = map(lambda mod: mod.predict(mels2014, 16, 1), [
    load_model( '{}/Training Onsets Model 63.98 64.00.hdf5'.format(modelFolder), compile=False),
    load_model('{}/Training Offsets Model 43.21 39.53.hdf5'.format(modelFolder), compile=False),
    load_model('{}/Training Volumes Model 98.72 96.95.hdf5'.format(modelFolder), compile=False)])
actProb2014 = load_model('{}/Training Actives Model 83.18 78.88.hdf5'.format(modelFolder),
                         compile=False).predict([mels2014, onProb2014, offProb2014], 16, 1)

Regarding RAM error during training, it is the same. There is a batch size parameter in my "TrainAndSave" function. Currently, for "velocity model" training it is 8. Reducing it to 4 will most probably help. However, training will be slower.

Oh, and by the way, after getting a memory error, you may have to shut down the template and restart it again, otherwise the memory error may not go away.

[epona7471]

Hi,

My laptop parameters are:
RAM: 16Gb
Processor: Intel Core i7-7700HQ CPU 2.80GHz
GPU: NVidia GeForce GTX 1050 Ti
  Dedicated GPU memory 4Gb
  Shared GPU memory 8Gb

I have not really deeply looked at my templates for a couple of years. But as I remember, I also had memory errors during testing. That is why I calculate precision/recall/F1-score for each single MAESTRO musical piece, and then print average values for each year (2004, 2006, etc.). And on the next cell with ROC-curve graphs, I make calculations only for 2014 part of MAESTRO.
Performing testing on the full dataset at once is impossible on my laptop with 16Gb RAM. Not sure about 25Gb in your case, but most probably it will not be enough either.

During testing, are you also getting ResourceExhaustedError, or something like the following:

MemoryError: Unable to allocate 15.1 GiB for an array with shape (28346, 626, 229) and data type float32

If yes, then reducing the batch size may help. For example, during testing, there is a batch size parameter in "model.predict". It is 32 in the following lines of code:

onProb, offProb, volProb = map(lambda mod: mod.predict(m, 32, 1), [onsetsModel, offsetsModel, volumesModel])
actProb = activesModel.predict([m, onProb, offProb], 32, 1)

And it is 16 in the next cell in the following block of code:

onProb2014, offProb2014, volProb2014 = map(lambda mod: mod.predict(mels2014, 16, 1), [
    load_model( '{}/Training Onsets Model 63.98 64.00.hdf5'.format(modelFolder), compile=False),
    load_model('{}/Training Offsets Model 43.21 39.53.hdf5'.format(modelFolder), compile=False),
    load_model('{}/Training Volumes Model 98.72 96.95.hdf5'.format(modelFolder), compile=False)])
actProb2014 = load_model('{}/Training Actives Model 83.18 78.88.hdf5'.format(modelFolder),
                         compile=False).predict([mels2014, onProb2014, offProb2014], 16, 1)

Regarding RAM error during training, it is the same. There is a batch size parameter in my "TrainAndSave" function. Currently, for "velocity model" training it is 8. Reducing it to 4 will most probably help. However, training will be slower.

Oh, and by the way, after getting a memory error, you may have to shut down the template and restart it again, otherwise the memory error may not go away.

Thank you for your kind reply. It also took me a time to repeat test again and again.

I have one more question, regarding reference 'Onset and Frames', there is no neural network for offset prediction.
Is there any reason why you added neural network for offset to this concatenation?
The result indicates that note with offset shows less accurate quality than note only model.

[BShakhovsky]

Actually, there is offset prediction in Magenta model. Maybe, it is not mentioned in earlier descriptions and not drawn in pictures, but it is there. If you download Magenta pre-trained checkpoint and run predictions, there will be 4 outputs: frames, onsets, offsets, velocities. Also, it can be seen that there is offset part in their python file:
https://github.com/magenta/magenta/blob/main/magenta/models/onsets_frames_transcription/model.py

Regarding your last question, you probably mean not two different kinds of models, but two accuracy metrics: "note" and "note with offsets". They both can be applied to the same model (with offset prediction part). "note" accuracy checks for onsets and ignores offsets, while "note with offsets" accuracy also checks for note durations. "note with offsets" accuracy is stricter, that is why its value will be smaller.