The update connections method requires memory which size is order of the square of neuron's size
Closed this issue · 3 comments
This is the bottleneck of memory for running huge size of simulations. We need to reconsider the update connections method.
(Created from #219 )
@fumik I assume you mean the ConnGrowth::updateSynapsesWeights() method, which copies a O(N^2) array to the GPU merely to update the synapse map entries. So, most of the values transferred are zero; some of those might be used to delete synapses.
On the CPU side, we can relatively straightforwardly replace CompleteMatrix objects with SparseMatrix objects, and then the size of the matrices involved will be O(S), where S is the number of synapses, rather than O(N^2). I think at one point in time that was the case (in ConnGrowth.cpp). This would reduce the CPU size footprint (assuming we ensured that the code didn't perform operations on the SparseMatrix objects that created N^2 zero entries). If we could ensure that the SparseMatrix weight object only had entries for synapses with non-zero weights, or synapses that had such weights before the update and are now zero (for deleted synapses), then I'm sure we could develop code to copy over a "synapse update table" that just has weights to be updated.
Now at the ConnGrowth::updateSynapsesWeights() method on GPU, we allocated a GPU memory, which size is O(N^2)/<# of GPU cores>.
The following is the memory usage of a 200*200 simulation.
Before allocation neurons and synapses structures.
used = 96.375 MB/GPU (0.84% of total GPU memory)
After allocation neurons and synapses structures.
used = 406.375 MB/GPU (3.5% of total GPU memory)
While updating synapses weights (before and after the fix)
used = 6512 MB (57% of total GPU memory) -> used 1934.375 MB (17%)
After creating SynapseIndexMap
used = 413.875 MB/GPU (3.6% of total GPU memory)
Though the ConnGrowth::updateSynapsesWeights() method is still be most memory consuming task, this fix slightly alleviate the problems. We need to revisit this issue and reconsider the new method mentioned above.