Startup with asynchronous components?
danielcompton opened this issue · 8 comments
I'm looking to use Component with a fairly large ClojureScript app. As part of the application startup, we need to create a websocket connection to the server, then issue some queries through that websocket before we can start using the app.
The component startup lifecycle seems to be built around making blocking calls to create stateful resources like database connections. This model doesn't really work in ClojureScript where you can't block for remote calls. Would you be interested in a PR that adds an async option for creating a system?
Asynchrony is available through other means, such as core.async go blocks and mutable Atoms.
Asynchrony through those means would infect the rest of the application and force it to dereference the atom, or do the work it needs to in a go block every time it needed a component (I think?). Having an asynchronous startup process would allow the application to treat async and sync started components the same after it is started.
Put another way, once a Clojure application is started, how a database driver wanted to be initialised, either sync or async shouldn't matter to the application.
Introducing asynchrony, in my experience, always affects the entire application. If the initialization procedure for some resource is asynchronous, then every piece of code which uses that resource must deal with the possibility that the initialization is not complete.
I am not aware of any approach that can completely hide asynchrony without the ability to block. If the system startup process were somehow made asynchronous, then application code would still have to deal with the fact that the system might not be completely started.
I'm requesting to kindly reconsider this request along with extensive problem illustration and a proposed APl.
Almost everything in JavaScript world doesn't start or stop synchronously.
Problem 1:
Assume the following scenario:
Component C depends on component D.
- In start, component D starts an external JavaScript API, which returns a promise P. component D assocs that promise to itself.
- After or during start component C calls a method on component D
- Because method on D doesn't know whether the external Javascript API is loaded, it needs await P before doing any work.
Awaiting one or multiple promises on the own started component, like descripted in step 3, is what you have to do in every method that you implement on a component in ClojureScript that has something async in start. This quickly becomes ceremony, like
(defn request-payment-handle [payment-component payment-request]
(let [payment-provider (<! (:payment-provider-promise payment-component))
;; this line in every method of payment-component namespace
]
;; do stuff with payment-provider
))
You can hide it with helpers and macros, but still it could be avoided.
Problem 2:
There is no way to await stop of the entire system in a reloaded workflow. For example, if you have a webserver component in your system that doesn't provide a synchronous stop method, you will have to write annoying code like
(defonce still-running (atom nil))
(defrecord Server [config]
component/Lifecycle
(start [this]
(assoc this
:server
(go
;; await previous stop
(when-let [p @still-running]
(<! p))
(reset! still-running (promise-chan))
(<! (start-server config)))))
(stop [this]
(go
(<! (stop-server config))
(close! @still-running))
this))
The above workaround would have to be further extended to work in a multi-instance setup. It is annoying boilerplate, and it can get more complicated. In CLJS projects, I found myself having to write synchronization boilerplate like this a lot, to make a reloaded workflow (reset) happen.
This could all be avoided by supporting async start and stop. Working with component on async/single-threaded hosts would become much more pleasant. Here is a rough sketch how this could be implemented:
- Optional
LifecycleAsyncprotocol. start and stop methods are passed adonefunction which a user can call with the a started component or to signal an error. This design is intended to avoid component depending on libs like core.async (which would allow for a fancier API of course).
(defrecord ServerAPI [config]
component/LifecycleAsync
(start-async [this done]
(.start (.jsWeb/Server. (:port config))
(fn [server error]
(if err
(done nil err)
(done
(assoc this :server))))))
(stop-async [this done]
(.stop (:server this)
(fn [success error]
(if err
(done nil err)
(done this))))))
start-system-asyncandstop-system-asyncmethods
(start-system-async my-system (fn [started-system error] ))
(stop-system-async my-system (fn [stopped-system error] ))
They would fall back to the synchronous start/stop methods on components that don't implement LifecycleAsync
This would restore the idea that components are passed started dependencies, which right now is always a lie, because almost nothing in JavaScript starts synchronously.
A "reset" / reloaded workflow for this would be a bit tricky, as it has to support queuing calls to reset and awaiting stop and start, similar to my previous problem illustration with a webserver. However it could be written once and for all for all component projects, and all previously illustrated synchronization problems would be eliminated on the system level once and for all.
Hi @lgrapenthin, thanks for working on this! I think it's an interesting idea, and something worth exploring. However, it's also a departure from the design of the Component library, which a lot of production systems depend on in its current form. I don't want to add anything new — especially a new API — until it's been had a chance to grow and be tested in other projects.
As you point out, it is tricky to make a reloaded-style workflow in an asynchronous world. I don't have any idea how to do it. I originally designed the Component library without any thought for ClojureScript. I regret porting it to ClojureScript at all, since the synchronous API is so obviously a poor fit for that environment.
Because this new async API is so different from the original Component API, I think it would make sense to explore it in an independent library, where it's easier to make changes rapidly without risk of breaking too many downstream projects. I don't have bandwidth to pursue this project myself, but I would be interested to learn about any discoveries made along the way. If that evolves into a stable API that has been tested in multiple production projects, and there is a compelling benefit to be had from deeper integration with this library, then I will be happy to look at including it. Best wishes and good luck!
@stuartsierra Hi Stuart, thanks for your thoughtful reply. I took a shot at this a week ago. After a first naive port of the loader to async, I realized that in comparison to assoc'ing promises and channels, asynchronously awaiting each component start is slower - which is not a good fit for web apps. So then I implemented a loader that would load dependencies in parallel. It was a bit tricky without an async framework. This might be interesting for Clojure projects as well, as a parallel loader could improve startup time there as well. Still it is faster to not await loading dependencies that are not required in start, i. e. a UI might depend on a payment API, but still be able to render without it. On the other hand, a synchronized teardown is really useful for reloaded workflow.
I invited you to a private repo. I would appreciate any kind of input. There are three issues with some contemplations. Thanks for component, it has served me well since its incarnation.
Hey @lgrapenthin did anything ever come of those efforts? I'm currently exploring using the component model in the CLJS context. Curious what you came up with.