Log files and multiprocessing

[mstimberg]

Our current logging infrastructure is not compatible with running multiple processes via Python's multiprocessing module. I guess working on examples that use multiprocessing prompted @schmitts' question about switching off the log file (#1407)?
A minor issue is that the log file is not very useful with multiple processes, since all processes write in no specific order. A bigger issue is that our log file rotation will raise an error if the file becomes big enough to be rotated – several processes try to move/delete the file, leading to a FileNotFoundError. I don't think there's a need to deal with this in a perfect way (e.g. by writing a multiprocess-safe logging handler, moving the log rotation into a single process, or similar), but we should at least mention this issue in the documentation. Or, given that few users actually find such details in the documentation, we could raise a warning in the logging code when 1) a file log is used and 2) it is accessed from multiple processes. A downside of this warning would be that I don't see an easy way to only raise it once, instead of once for every process launched by multiprocessing. But I guess this is ok, since we really want the user to do something about it.
There are three workarounds that we can recommend to users:

1. Use a preference file to switch off file logging
2. Set BrianLogger.tmp_log to None to switch off file logging in an ad-hoc way
3. Call BrianLogger.initialize() at the beginning of each spawned process, which will make it use its own log file

[schmitts]

Yes, I was also encountering the FileNotFoundError error and was using a preference file to switch off the logging. This seems to be the best solution (with #1217 one would not need the preference file).

I would not advertise 2. It depends on an implementation detail.

I will try out 3. No opinion atm.

[thesamovar]

Can we detect if we're being run from multiprocessing and automatically switch off logging if so? i.e. only the original launch would log.

[mstimberg]

We could store the original process id when we initialize the logger, and not log when the current process id does not match the original one, I guess. Switching the logger off completely in this case should be quite straightforward, but we could also only switch off the file log with a bit of additional code. Maybe best to only switch off the file logging to not get rid of important warnings? Should we still raise a warning if we do it, to make users aware? Not sure whether this would be just annoying or helpful... Maybe rather no warning and mention workaround 3 from above in the docs – I'm not sure that many users will miss log files, after all.

As a tangential aside: I think we should raise the default file log level from DIAGNOSTIC to DEBUG, we are writing so much stuff to these files by default...

[mstimberg]

To summarize my somewhat convoluted comment above, I think my preferred solution would be:

1. Switch off logging to files but keep logging to console when os.getpid() does not match original pid (stored when BrianLogger.initialize() is called). Do not issue any warning when doing this.
2. Document the above, and mention that each spawned process can call BrianLogger.initialize() to get its own process-specific log file
3. Raise the default file log level to DEBUG

What do you think about this solution @thesamovar and @schmitts ?

[schmitts]

Sounds good to me!

[thesamovar]

Yes, yes, yes!