Finer object composition for the controller class
Closed this issue · 7 comments
In crew
, every R6
controller object contains 2 smaller R6
objects:
- A router to keep track of
mirai::daemons()
and themirai
dispatcher, and - A launcher that knows how to create, monitor, and terminate
mirai
servers on the user's platform of choice. A launcher plugin is anR6
subclass that inherits from the abstract launcher class.
That leaves a lot for the controller to do: namely,
A. Manage mirai
task objects (e.g. controller$collect()
), and
B. Auto-scale mirai
servers (`controller$scale()).
Both A and B are highly nontrivial, and if @shikokuchuo's promising idea from shikokuchuo/mirai#60 (comment) comes to fruition, then each will run on different processes. So from crew
's side, it makes sense to create R6
classes for each: a "schedule" for A and a "regulator" for B. The regulator will contain the launcher, and it will be shipped to the dispatcher process in shikokuchuo/mirai#60 (comment) to do auto-scaling given a very simple set of inputs.
Currently, the object composition structure looks like this:
controller
├── launcher
└── router
My goal for the structure is:
controller
├── schedule
├── regulator
│ └── launcher
└── router
One nice simplification would be to have the launcher call mirai::saisei()
. If auto-scaling becomes possible on the mirai
dispatcher, a dispatcher-side version of saisei()
may be needed, as the current saisei()
seems to be designed for the client only.
Launcher-side saisei()
and poll()
work great and simplify things. In fact, I don't think crew
even needs a "regulator" class. The launcher itself can auto-scale if I calculate demand differently.
Also, I am thinking of a couple more changes (unfortunately breaking ones for launcher plugins, but crew
is still new enough for that):
- Rename "router" to "client".
- Condense the
launcher
,worker
, andinstance
arguments oflaunch_worker()
down to a singlename
argument with the worker name already fully formed.
Also should bring back seconds_interval
to launchers to prepare for integration with the mirai
dispatcher.
In branch 84
, I simplified the internals a whole lot. Whether or not crew
will be able to integrate with the mirai
dispatcher (which will require a little more work on my end, but nothing painful at this point), I am happy with all the cleanup in this heavy refactor.
Now I need to debug and test:
- Rd examples
- README examples
- Vignette examples
- Launcher tests
- interactive tests
- Unit tests
- throughput tests
-
crew.cluster
integration -
targets
integration
Condense the launcher, worker, and instance arguments of launch_worker() down to a single name argument with the worker name already fully formed.
No, this didn't work. crew.cluster
relies on having access to each of these pieces separately. Users can construct job names on their own.
Best of both worlds: supply both the job name and the pieces.