How to let multiple independent processes to log into the same place?
Opened this issue · 2 comments
I'm running multiple python scripts at the same time, and deliberately want them to log into the same place.
I suppose this is doable as the module supports multiprocessing, but not sure what is the best practice of doing this.
log into the same place
I assume you mean send log into the same logger process. Then that logger process will write the log to its final destination.
Then yes, it is doable.
It includes two parts that involve TCP/UDP logger:
(1) Logger server: the script that starts the TCP server with a predefined port.
(2) Logger clients: other scripts that use TCP/UDP logger and send logs to the server.
For (1), you just start logger_tt with use_multiprocessing=True, port=YOUR_PORT, and we are done. It will start the logger server usually. No need to really spam any child process in that process.
For (2), you also need to start your scripts with use_multiprocessing=True, port=YOUR_PORT. But this time, you need a way to tell logger_tt not to start the logger server, just replace the log handlers with the TCP handler.
Considering the above scenario, I think we need to add a new param to setup_logging, pass it down to LogConfig, and then into _replace_with_socket_handler method to tell it whether to start the logger server or not.
It includes two parts that involve TCP/UDP logger:
(1) Logger server: the script that starts the TCP server with a predefined port.
(2) Logger clients: other scripts that use TCP/UDP logger and send logs to the server.
I wrote one just like this yesterday. and the trick i used is adding --multiprocessing-fork as the first argv for all the "client", so that they are considered as subprocesses by in_main_process.
It worked so far for me, but may have some side effects if other module also need to check if it's main process.
I think maybe we can add an _server attribute which will defaults to the result of in_main_process but can be overridden by input arguments.