idefix-code/idefix

BUG: IdefixArray defined in setup.cpp's global scope are not properly deallocated at the end of the job

Closed this issue · 2 comments

Describe the issue:

Can I define and use a global IdefixArray in setup.cpp in case I want to use the same array in, say, InitFlow and MyBoundariesFunc without declaring and initialising it in each function? Because when I do so (declaring it as a global variable in setup.cpp outside a function, initialising it in the Setup function and then using it in whatever other function I want), it seems to work fine but at the very end (after the job completed successfully) there seems to be a deallocation error. I am running on CPU right now.
I want to do that because I need to use the same IdefixArray in a user-defined source term function (which is called at each time step I guess) and I don't want to declare it and initialise it each time within this function because I think that it would be a huge bottleneck for the code performance. The specific IdefixArray I want to use coutains value of the (vector) spherical harmonics up to \ell and m for which analytical expression is not easily available (but that the library shtns is able to compute).

Error message:

Main: Job completed successfully.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Kokkos allocation "MyAwesomeArray" is being deallocated after Kokkos::finalize was called

Traceback functionality not available

Aborted

runtime information:

first noticed with old version of Idefix, but reproduced with the latest commit of v2.0 branch (commit f02fb50)

Thank you for taking the time to report this problem.
I think it is reasonable to want this to be supported, however I currently don't see any simple way to do it; Here's what I can grasp:
Kokkos::finalize() is responsible for cleaning up memory before the program exits. As per Kokkos' documentation, it must be called before MPI_Finalize, so we cannot really delay it in any way (it must be called in the main function)
The problem is that any IdefixArray living in the global scope of a module (say setup.cpp) will be destroyed after the main function returns.

The only way (I see) we can allow Kokkos to clean up all arrays is to require they all be data members of objects that live in the scope of main. mysetup would seem like the ideal candidate here but:

  • mysetup isn't accessible from anywhere outside the main function (making it useless as a global data container)
  • you'd still need to modify the class definitions in order to add data members.

I don't want to declare it and initialise it each time within this function because I think that it would be a huge bottleneck for the code performance.

For completion: you're guessing correctly. (De)Allocating arrays of any size is extremely costly so we definitely don't want to do it for large arrays at every time step.

glesur commented

This is an expected behaviour. Arrays (actually memory chunks) are automatically dealocated when all of the IdefixArrays refereing to that memory chunk have been deleted. This deletion happens either implicitly (by a closing scope) in which case the objects contained in the scope are all deleted automatically, or explicitly (through a new/delete pair).

As @neutrinoceros pointed out, If you define an IdefixArray in the global scope, it is deleted when the program terminates. Hence deallocation should happen then. Except that normally we need to call Kokkos::finalize before the program terminates and this finalize should be done once all of the Kokkos objects have been deleted (this includes IdefixArray. So no matter what, IdefixArray in the global scope will always create problems when Kokkos:finalize is called.

The way to treat this is to explicitely delete the object when you don't need it anymore. The cleanest way to do this for a setup is to define a "container" class, containing all of the arrays you will need in the global scope, and just have a global pointer to an instance of this class, that you eventually delete (and which deletes all of the arrays it contains automatically). More explicitely:

  • start with a declaration of a class container (that I name MyGlobalClass in this example) and a global pointer to a class instance (note that you can put as many arrays as you want in the class)
// Class declaration
class MyGlobalClass {
 public: 
  // Class constructor
  MyGlobalClass(DataBlock &data) {
   //allocate some memory for the array the class contains
    this->array1 = IdefixArray3D<real>("MyAwesomeArray",data.np_tot[KDIR], data.np_tot[JDIR], data.np_tot[IDIR]);
  }
  
  // array1, member of the class
  IdefixArray3D<real> array1;
};

// A global class instance named "myGlobals"
MyGlobalClass *myGlobals;
  • next initialise your global object in the Setup constructor (this will aumatically allocate the array it contains thanks to the class constructor we have defined):
Setup::Setup(....) {
   ...
   myGlobals = new MyGlobalClass(data);
   ...
}
  • use your array when you need it:
MyXXXXFunction(....) {
  // Shallow copy the global array
  IdefixArray3D<real> array = myGlobals->array1;
  // Do stuff
  ....
}
  • and finally, to avoid the error message "Kokkos allocation "MyAwesomeArray" is being deallocated after Kokkos::finalize was called", delete the object on exit in the Setup destructor
Setup::~Setup(....) {
   ...
   delete myGlobals;
   ...
}

If you just need one single global IdefixArray, then you can just define a pointer to an IdefixArray in the global scope and use a new/delete in the Setup constructor/destructor. However, usually one wants several arrays, in which case the container class approach presented above is cleaner.