/Vunsq

Functional Video Sequencer

Primary LanguageQML

Vunsq

Functional Video Sequencer

Like a mod-tracker, but:

  • For timing and blending 2D video instead of sound, and
  • Using generative, pixel-shading functions for the 'samples' instead of pre-recorded videos

Core Concepts and Terminology

An Effect is a function like a pixel shader: it gets fed an x coordinate, a time, and (optionally) custom arguments, and it produces RGBA values for an entire light strand. You can have a maximum of 128 different Effects.

An Event is an instance of an Effect. It specifies a time to start, a the strands it affects, and optional custom arguments. Events reference Effects by numeric identifier.

A Presentation is the entire ‘movie'. It specifies a BPM it is associated with, and an ideal reference song to play with it. A Presentation has a Timeline that groups together all Events to display. Presentations can be played by the runtime along with a different song at a different BPM, speeding them up or slowing them down.

Note: Effects are code in the host runtime, and not stored within a Presentation.

JSON Example

{ "bpm" : 100.32, "length" : 237483, "media" : "CHVRCHES-LeaveATrace.m4a", "timeline" : [ [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":2, "speed":2.5 }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":2, "speed":2.5 }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":2, "speed":2.5 }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[255,0,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":0, "start":0 }, { "effect":1, "start":598, "args":[0,255,0] }], [{ "effect":2, "speed":2.5 }] ] }

Binary Format

All multi-byte numbers are stored in little-endian format.

Presentation

bytes field
4 Presentation BPM (float)
4 Presentation duration (uint32)
1 Characters in Media URI (uint8)
~ Media URI (UTF-8)
~ Timeline Index
~ Timeline

Timeline Index

bytes field
1 Strand Count (uint8)
4 Offset to Strand 1 Timeline (uint32)
2 Byte count of Strand 1 Timeline (uint16)
4 Offset to Strand 2 Timeline (uint32)
2 Byte count of Strand 2 Timeline (uint16)
4 Offset to Strand … Timeline (uint32)
2 Byte count of Strand … Timeline (uint16)
~ (repeat for all strands)

Timeline

bytes field
4 Strand 1 Event Count (uint32)
~ Event 1
~ Event 2
~ Event …
4 Strand 2 Event Count (uint32)
~ Event 1
~ Event 2
~ Event …
4 Strand … Event Count (uint32)
~ Event 1
~ Event 2
~ Event …

Event

bytes field
1 Effect# (uint8; 128-255 only)
4 Start (uint32 ms)
4 Speed (float)
1 Arg Count (uint8)
1 Arg 1 (uint8)
1 Arg 2 (uint8)
1 Arg … (uint8)