matsim-vsp/parallel_qsim_rust

Reduce memory usage.

Closed this issue · 0 comments

Currently each MPI-Process loads the network and the population file. If we have a lot of slots on a single machine, we might run out of RAM, also reading from disk becomes a bottle neck then. (Even on HLRN we run out of memory if we assign the full 192 slots available per machine)

First idea:

One process does the loading.

  1. Load Network
  2. Load Population
  3. Do partition stuff and id mapping
  4. Broadcast network and population partitions, as well as id-mappings to other processes.

Refinement:
Split up loading of Network and Population. Each of the loading processes can also do the id mapping and broadcast this to the other nodes.