A simple shell command manager. Used for running parallel simulations.
First install the package with pip
:
pip install parallel_manager
To run the manager, we will need to wrap the manager initialization and request creations all in a async function like this:
from parallel_manager.manager import BaseShellManager
from parallel_manager.workerGroup import ShellWorkerGroup
import logging
import asyncio
async def Main():
# Init the manager
simpleShellWorkergroup = ShellWorkerGroup("simpleShellWorkergroup",
logging.getLogger(),
"./log",
10)
simpleShellManager = BaseShellManager("simpleShellManager")
simpleShellManager.add_workergroup("shell", simpleShellWorkergroup)
await simpleShellManager.init()
# Adding tasks
# Here we just run 100 echo
for i in range(100):
simpleShellManager.add_shell_request(f"echoing loop-{i}", f"echo {i}")
# Wait for the manager to finish
await simpleShellManager.done()
Then we simply use asyncio.run()
to run the above function:
asyncio.run(Main())
The manager will run at most 10
processes at a given time and will terminate when all 100
requests are done.
Full example is at here
- Worker
- Receive request from queue and process it
- WorkerGroup
- Manage workers of same class
- Collect basic stats from workers
- Manager
- Might host various workergroups with different worker type
- Support various ways to add tasks
- Control workergroups/workers
- Various middleware/plugin to the BaseManager to implement other features like http web monitoring and task submission
- Manager statistics plugin
- Manager summary when all tasks are done
- Manager prefix on creation
- Timeout support
- Status monitor, progress monitor
- Dump failed task/able to rerun
- Dump unfinished tasks when script terminates
- Kill ongoing shells when program abort (
atexit
)
- Thread-mode support (completely non-blocking)
- Use a separate thread to run the workers
- Allow to submit task at any time
- Http monitor
- Interactive control for status monitoring and changing configuration dynamically, treat worker as a server that can submit task to, like a local slurm control
-
PythonFunctionWorker
class (execute python function calls) and other worker class as well - Standardize plugin format for each class
- For instance, http server monitor could be a plugin that can be added to any manager class.