Create an adaptive music player for Role Playing Games. This will be controlled by the DM/GM and will be able to stream music that is appropriate to the current game state to a client application. As this type of game is often played online, two client applications are planned, one for the DM and one for the other players to receive the music
MVP
- A composer should be able to provide audio assets in the correct folder structure and with appropriate metadata and have this scanned into the system
- A user should be able to stream music from a server - to be implemented
- A user should be able to enter current game context and hear music that is relevant
- A user should be able to adjust the intensity and darkness of the music
The project is currently able to supply music locally responding to input from the frontend, but the streaming is not implemented so the music will only play on the server. A workaround for this has been tested using an icecast radio server but the latency introduced by this is too great to make it worthwhile. A solution using WebRTC is probably how this will be implemented - LiveKit has been investigated and looks like a good candidate.
The ability for a composer to add music is implemented and relies on the correct directory structure, parameters encoded in the filenames and json files containing other data. When the Java server is started the directories are scanned and the data is added to the MongoDB database. A further explanation of the way the music files need to be formatted and presented is explained below.
The backend server is currently undergoing a major refactor to make the current state available to the frontend and is a little fragile in places. The initial sprint was two weeks and the main goal was to have something to show a demonstration of the functionality. The frontend is able to connect over a websocket and pass parameters to the backend which responds accordingly but the feedback to the frontend will be implemented through the refactor. Due to this the available locations are currently hardcoded in the frontend, these will be made more dynamic in the future.
- Clone this repo!
- Use this link to get demo audio files.
- Unzip that into the RPGMusicServer/src/resources/static folder
- Go to the rpg-music-client folder and use
npm iin the terminal to install the frontend dependencies - Then run
npm startto run the client and go to http://localhost:3000 in your browser. - Open up the RPGMusicServer folder in your Java IDE (I like intellij)
- Build and run the RpgMusicServerApplication.java
- Hit reconnect on the client and hit the buttons to play some music!
Demo audio files are provided but here is a quick rundown of how the files are used and structured if you would like to make your own. Audio files should be placed in the resources/static folder in the server folder. The directory structure is shown below and there will follow a further explanation of what each level means.
/tunes
/tune
/movement
/section
section_data.json
/part
/museme.wav
The top level of this directory structure is tunes, and this surprisingly contains a folder for each tune! A tune here refers to the music for a particular game context which is usually a location, but could also be a current game event such as combat. We'll use an example location of an Enchanted Forest to demonstrate this. So the directory structure would look like this
/tunes
/Enchanted_Forest
The next level down is the movement. This refers to a musical movement which itself could be made up of multiple sections. For our Enchanted Forest tune we have two movements. The first is Intro, which is where the DM would be giving some flavour text and introducing the new location. The second movement is the main Story loop which will play throughout the adventure time in the forest. The folders containing each of the movements are prefixed with a zero based index followed by an _ to maintain their order.
/tunes
/Enchanted_Forest
/0_Intro
/1_Story
Next we have each of the sections which make up the movements. Here the intro movement is made up of two sections "main" and "out". The main section is a loop which, when triggered, will move to the out section ready to progress to the next movement. The Story movement contains a single "main" loop which will play until it is either stopped or a new tune is triggered.
/tunes
/Enchanted_Forest
/0_Intro
/main
/out
/1_Story
/main
Each of these sections must contain a section_data.json file which has the required musical data in it. Here's an exmple of this
{
"order": 0, // The sections' order within the movement
"numBars": 4, // Length of the section in musical bars
"numBeats": 4, // Number of beats in the bar (Top of a time signature)
"beatValue": 4, // Value of each beat (bottom of a time signature)
"bpm": 110, // Tempo - beats per minute *** Double this if beat value is 8
"preRendered": false, // If the section is a single rendered file or not
"loop": true, // Is this a looping section
"transitionType": "END" // When this section transitions to the next END is end of section,
// NEXT_BAR transitions on the next bar
}
We will now just focus on the Intro movement to avoid needless repetition.
Sections come in two different types, RenderedSections and AdaptiveSections. RenderedSection is the simplest so that will be demonstrated first.
In this case the /0_Intro/out section is a RenderedSection. This means that it contains a single audio file that is premixed and rendered.
To do this the audio file is placed in a /MASTER/ directory. The audio file must be CD quality or more specifically, 44100KHz 16bit signed little-endian wav.
(This is the case for all audio files in this project)
/tunes/
Enchanted_Forest/
/0_Intro
/main
/out
section_data.json
/MASTER
/master
/Master.wav
The main section is an AdaptiveSection which is made up of parts that are dynamically selected and pieced together. These
are grouped in each section by defined musical types. The currently allowed part musical types are
MASTER, MELODY, PAD, BASS, HIGH_PERC, MID_PERC, LOW_PERC, CYMBAL
The main section example we are using here only uses MELODY, PAD, and BASS , giving the following directory structure within that section.
/main
/MELODY
/PAD
/BASS
/section_data.json
Parts are placed within each of these categories. These are directories with a name following the format of instrument_i1_d2.
Here the name of the instrument can be anything, i1 indicates that this part has an intensity threshold of 1 and d2 indicates
a darkness threshold of 2. These values can be in the range of 1-5 and can be omitted entirely to have no threshold set. These values are
used by the system to decide which parts are appropriate for the current game state. Here is what the MELODY folder could look like
/MELODY
/flute_i1_d3
/harp_d4
Within each of these part folders, the individual 'Musemes' (Like phonemes but for music - individual files that can be pieced together)
are placed. There can be any number of these with the file name format of name_b2_l2. It is common for the name to just be a number. The b2
part indicates that the file should start on bar 2. If there are multiple potential start bars the format b1&3 would indicate that bar 1 or 3
are appropriate. A further value of bx can be used to indicate that any bar is an appropriate start bar. the l2 part indicates that
the musemes length is 2 bars. No further musemes will be selected in this part until the museme has ended. Both of these values can be ommited
and the museme will default to 1 as the start bar and it's length being the length of the section it belongs to.
/MELODY
/flute_i1_d3
1_b1_l2.wav
2_b3_l2.wav
/harp_d4
1.wav