IBM/adaptive-federated-learning

It is getting memory error with the MNIST dataset.

rohnitt opened this issue · 4 comments

rohnitt commented

While I am running the server with the MNIST dataset, it is getting memory error. The stacktrace is mentioned below. Please help me fixing this.
Traceback (most recent call last):
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\server.py", line 25, in
train_image, train_label, test_image, test_label, train_label_orig = get_data(dataset, total_data, dataset_file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\data_reader.py", line 30, in get_data
train_image, train_label = mnist_extract(start_index_train, total_data_train, True, dataset_file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\mnist_extractor.py", line 54, in mnist_extract
return mnist_extract_samples(sample_list, is_train, file_path)
File "C:\Users\Rohit\AppData\Local\Programs\Python\Python37\adaptive-federated-learning\data_reader\mnist_extractor.py", line 38, in mnist_extract_samples
x = np.array(list(f_images.read(rows_im * cols_im)))/255.0
MemoryError

shiqiangw commented

Are you running this on some device with a really small memory size?

rohnitt commented
shiqiangw commented

Not sure... A few years back, we trained the SVM model with MNIST on Raspberry Pi v3 devices that only have 1GB RAM. The CNN model might be too big (or too slow) for Raspberry Pi v3 though. Your devices seem to be more powerful and have more RAM.

rohnitt commented