Suggestions for triggering events on state change?
Closed this issue · 2 comments
Many systems (e.g. the keyboard visualizer linked from the readme) are well-represented as functions of state. However, other systems require you to trigger effects as events change, e.g. sending midi note on/note off events. I've spent a few hours playing around with dsim, with the hope that it might form a suitable foundation for a realtime audio event system. Unfortunately, I haven't figured out a good way to think about "triggering" things as state changes over time. Do you have any suggestions for how one might think about using dsim in this manner? Does this seem like a reasonable use-case for dsim?
Hi,
Thanks for your feedback! I would probably need to know more about what you are building, but I'll try to provide you with a general answer regarding side effects. First, a few words about the opposite of what you want in order to better understand the link between simulated time and real time (time in a context Vs. time in the real world).
That keyboard visualizer operates in two modes. In the "offline" mode, it reads a MIDI file and extracts all needed MIDI events. Once this is done, everything is pure. In the "online" mode, the context is stored in an atom and MIDI events are received live. The neat thing is that thanks to DSim, they both share 99% of the code.
So, in the online mode, MIDI events are side effects. Those side effects are translated into meaningful events in our system. Here, raw MIDI events are translated into events creating flows (typically finite ones), something like:
(fn on-midi-message [timestamp msg]
(swap! *ctx
(fn add-color-transition [ctx]
(dsim/e-conj ctx
[timestamp]
[:piano-key (:note msg) :color]
(dsim/finite 2000
(fn compute-color [ctx] ...))))))Note I schedule the creation of a the finite flow (ie. gradual transition of the color of a piano key), I don't compute anything else because I don't need to.
Other kind of side effects require you to know the exact state at a precise moment. Those side effects, happening at some point in time in the real world, dictate the current simulated point in time in the system (::ptime). When a frame is drawn, I have to know what to draw, hence run all events up to the timestamp of that frame. That would look like:
;; When provided with a context, returns a lazy sequence of all computed points in time.
;; We can navigate this sequence and stop when we need.
(def lazy-run
(dsim/historic (dsim/ptime-engine)))
;; Besides executing everything up to when we need to draw, we sample (compute) all flows
;; so that we know their exact state at the moment of drawing.
(fn on-frame [timestamp]
(let [ctx-2 (swap! *ctx
(fn update [ctx]
(last (take-while #(<= (dsim/next-ptime %)
timestamp)
(lazy-run (dsim/e-conj ctx
[timestamp]
nil
dsim/sample))))))]
(draw ctx-2)))All that was about side effects impacting our context. I wanted to insist on the fact that the context will not move through its simulated time until we move it ourselves.
What about the opposite, when the context satisfies some condition, do a side effect? You have to think about that link between simulated time and real time. A side effect could be just another event in the context. Then it will execute only when we run the context and reach that ptime, the moment this side effectful event is scheduled in our simulated time.
What if after reaching a condition in our context, we want to schedule a side effect for later IN THE REAL WORLD? Such as "when this and that, send a MIDI Note-On message after 2000 real milliseconds". Then you would use in addition an external timer, something that is indeed capable of waiting 2000 milliseconds in the real world before executing a function. There are a few options, core.async is even one, even (Thread/sleep) is a poor's man solution.
However, for real time audio, typically, timing in the real world has to be precise. You'll have to do some research. For MIDI, I use a Sequencer object because I know it does lower-level stuff that will probably result in more accurate timing than what I would write on the go: Sequencer. I use both DSim for computing graphics (+ other stuff) and that Sequencer (from the javax.sound.midi package) for live audio stuff.
I hope it helps, feel free to provide an more concrete example.
Thanks for such a thorough answer! It really clarified a number of dsim concepts for me. on-midi-message and on-frame are particularly helpful illustrations, as they very clearly explain what I assume will prove to be fairly common online use-cases - namely, updating the simulation on events and on a schedule. (I imagine others might find it helpful if you included these and other similar examples in the readme.)
What about the opposite, when the context satisfies some condition, do a side effect?
and
What if after reaching a condition in our context, we want to schedule a side effect for later IN THE REAL WORLD?
These describe my use case, which does seem a bit trickier to conceptualize than updating the simulation itself on external inputs. Specifically, I'm looking to construct an environment where various forms of realtime musical expression are possible. For example, turning an encoder might modify the value of some musical parameter (now or in the future), e.g. a "pitch bend" MIDI CC message. (This seems like a potential usecase for dsim's sampling tools, actually.) Or perhaps a user might schedule a (finite or infinite) loop, of which some aspect may be similarly parameterized. Or perhaps hitting a note on a drum pad fires a MIDI Note On event, and schedules a Note Off event for 800ms in the future. You get the gist.
Then it will execute only when we run the context and reach that ptime, the moment this side effectful event is scheduled in our simulated time.
This is probably the clue that I need to meditate on. I'll probably end up making an event loop that looks something like this -
On an external input (e.g. a change in an encoder value),
- Update a stateful
ctxatom using standarddsimmechanisms - Cancel any timer cancel callbacks (from step 4) between now and some
buffer, e.g.(+ now 100) - Perform a range query on
::dsim/events, grabbing:
a. the first event after now, and
b. all events between now andbuffer - Schedule timers for the events in 3, and store their cancel callbacks in
ctxto be used in successive iterations of step2
Does that make sense? As a rough sketch, at least?
On a personal note, the timing of your publishing of this library is quite serendipitous - just a week ago, I was fumbling around in the dark with some of these ideas, and I sketched out something conceptually similar (at least at a high level). Here's a quick copy/paste of what I was working on:
(ns mm.events
(:require [kdtree :as kdtree]))
(def schedule-default
;; Map of ticks to sets of event symbols
{:ticks->event-id-sets {}
;; Map of events to the tick at which they're scheduled
:event-ids->ticks {}
;; Map of event symbol to event
:events {}
;; kd-tree containing all ticks with scheduled events
:ticks (kdtree/build-tree [[-1]])})
(def schedule (atom schedule-default))
(defn reschedule-event
"Set the time of an event on the schedule.
The event needs an id and a tick.
The event should already be scheduled. Reschedule-event only alters the
event's time, not its other properties.
To set all properties of an event, use add-event."
[schedule event]
(let [id (:event/id event)
tick (:event/tick event)
old-tick (get-in schedule [:event-ids->ticks id])
schedule (if old-tick
(update-in schedule [:ticks->event-id-sets old-tick] disj id)
schedule)
event-set-empty? (empty? (get-in schedule [:ticks->event-id-sets old-tick]))
schedule (if event-set-empty?
(update schedule :ticks->event-id-sets dissoc old-tick)
schedule)
schedule (if (and event-set-empty? old-tick)
(update schedule :ticks kdtree/delete [old-tick])
schedule)]
(-> schedule
(assoc-in [:events id :event/tick] tick)
(assoc-in [:event-ids->ticks id] tick)
(update-in [:ticks] kdtree/insert [tick])
(update-in [:ticks->event-id-sets tick] #(if % (conj % id) #{id})))))
(defn add-event
"Add a new event to the schedule.
The event needs an id."
[schedule event]
(let [id (:event/id event)]
(-> schedule
(assoc-in [:events id] event)
(reschedule-event event))))
(defn reset-schedule
"Removes all events from the schedule."
[]
(swap! schedule (constantly schedule-default)))
(defn remove-event
"Removes an event from the schedule.
The event needs an id."
[m event]
(let [id (:event/id event)
old-event (get-in m [:events id])]
(if old-event
(let [old-tick (get-in m [:event-ids->ticks id])
m (update m :events dissoc id)
;; TODO: Only delete if ticks are empty
m (update m :ticks kdtree/delete [old-tick])
m (update m :event-ids->ticks dissoc id)
m (update-in m [:ticks->event-id-sets old-tick] disj id)
event-set-empty? (empty? (get-in m [:ticks->event-id-sets old-tick]))]
(if event-set-empty?
(-> m
(update :ticks->event-id-sets dissoc old-tick)
(update :ticks kdtree/delete [old-tick]))
m))
m)))
;; ;; TODO: Efficient range query over :ticks->event-id-sets
(def e1 (gensym "e1"))
(def e2 (gensym "e2"))
(def e3 (gensym "e3"))
(reset-schedule)
;; NEXT: Sanity check removal
;; this isn't quite right yet
;; THEN: make events-in-range
(swap! schedule add-event {:event/id e1 :event/tick 10})
(swap! schedule add-event {:event/id e2 :event/tick 10})
(swap! schedule add-event {:event/id e3 :event/tick 10})
(swap! schedule reschedule-event {:event/id e1 :event/tick 20})
(swap! schedule reschedule-event {:event/id e2 :event/tick 30 :event/name "jake"})
(swap! schedule add-event {:event/id e2 :event/tick 3 :event/name "jake"})
(swap! schedule reschedule-event {:event/id e3 :event/tick 10})
(swap! schedule remove-event {:event/id e1})
(swap! schedule remove-event {:event/id e2})
(swap! schedule remove-event {:event/id e3})(The :ticks key doesn't quite work yet, and I didn't think to use a sorted map for :ticks->events-id-sets, which would have allowed me to remove my dependency on kdtree... Ah well.)
I love that moment when you discover that the idea that you've been chipping away at is a well-formed area of study! Anyway - it's awesome to see such a fleshed out implementation of the concept. There's a lot for me to dig into here.