audiojs/audio

alternative audio data structures / storing ops instead of applying immediately

dy opened this issue · 2 comments

dy commented

Following audiojs/audio-buffer-list#5.

The current API approach is covered by a lot of similar components, it is destined to insignificant competition and questionable value. The main blocker and drawback is the core - audio-buffer-list component, which does not bring a lot of value, compared to just storing linked audio-buffers.

Alternately, audio could be focused on storing editing-in-process, rather than data wrapper with linear API, similar to XRay RGA.

Principle

  • storing operations rather than applying them to data
    • 👍 no precision loss
    • 👍 faster insertion/removal
    • 👍 allows for collaborative editing
    • 👍 allows for faster re/adjusting params of applied control/envelope
    • 👎 possibly somewhat slower playback due to applied transforms stack, hopefully having heavy-duty fx is not a part of editing process
      • ! possibly compiling fx program dynamically, akin to regl
      • ! pre-rendering audio for faster playback
  • undo/redo history methods store operations, not full binary replica every step
  • branching - allows for alternatives

Pros

  • 👍 makes audio unique
  • 👍 makes it suitable for editors
dy commented

Reference structures:

In fact, git seems to be suitable for that too.

Note also that the class technically should allow to utilize any underlying technology: time series, STFT, formants, HPR, HPS SPS etc models (https://github.com/MTG/sms-tools/tree/master/software/models), wavelets etc.

👍 In the case of formants, for example, transforms are theoretically times faster than the time series.
👍 Abstract interface would discard sampleRate param and make Audio just a time-series data wrapper, with possibly even uncertain/irregular stops. We may want to engage a separate time-series structure for that, which seems to be broadly demanded, from animation/gradient/colormap stops to compact storing of observations.

dy commented

Lifecycle

  1. Initialize data model
  2. Input data source
    • Convert input data source to target model
  3. Modify data source
    • Create stack of modifiers/reducers/transforms
    • Modifiers can possibly be applied real-time
  4. Play data source
    • Apply stack of transforms, play / apply transforms per-buffer
  5. Get stats
    • Should model include stat params straight ahead?
  6. Output data source
    • Apply stack of transforms, output

Plan

  • Collect set of concerns, use-cases of responsibility
  • Come up with ideal API covering all these cases
  • Create baseline/edge/real tests for the cases
  • Fix tests

Stores

  • time-series store
  • web-assembly store
  • stft store
  • harmonic model + residual store
  • formants store
  • wavelets store
  • see ref for other stores (Bercelona Institute)