MATOS is a software package for interactive multi-agent sound generation specifically designed for use with tangible user interfaces. As part of this project, MATOS will be integrated with the AudioCube hardware design of Robert Van Rooyen (see the AudioCube Design Document). MATOS is developed using C++ for handling agent communication, logic and reasoning; while PureData with LibPd is used for sound generation, with C++ integration using LibPd. This design ensures that MATOS is cross platform and can be built on any system with the correct libraries installed using CMake. Additionally using PureData allows for robust and flexible sound design even for composers not familiar with C++.
- Install above dependencies
- CMake currently assumes that dependencies are in
/usr/local/include
and their libraries are in/usr/local/lib
- Update CMakeLists.txt if installations are located elsewhere
- CMake currently assumes that dependencies are in
- Clone MATOS repository from GitHub:
git clone https://github.com/fortjohnson/MATOS.git
- Build MATOS
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - Unix Makefiles"
make
- Run the application
- Edit
neighbors.txt
with the IP addresses and osc listening ports of other MATOS agents ./MATO -i <agent id> -o <osc port> -p <PD patch location> -n <neighbors file> -s <number of states>
- agent id: any unique integer
- osc port: port for MATOS to listen for OSC messages from agents (default = 7000)
- the OSC port sensor input is the agent port number + 1000 (default = 8000)
- PD patch location: path to the PD patch for sound generation (default = pd/main.pd)
- neighbors file: path to the file containing connection information for other MATOS agents
- number of states: in the
states
folder - Example:
./MATO -i 0 -o 7000 -p ../pd/main.pd -n ../neighbors.txt -s 4
- Edit
The demo
folder contains the necessary files for a demo of MATOS with 4 agents. After successfully building and
running MATOS on each computer
- Browse to the
demo
folder - Select one of the 4 demo folders and open the corresponding
pd
folder - Copy the
samples
andstates
folders andmain.pd
- Paste into the
pd
folder of the rootMATO
folder to replace the existing the existing files - Now from the
pd
folder, copy both thestates
andsamples
folders - Paste in the root
MATO
folder- (this is a workaround that I'm hoping to address in the future)
- Edit the
neighbors.txt
file to configure the agent with a list of neighbors- Each line represents one neighbor agent.
- The first entry in the file is the agent id
- The second entry is the agent host's IP address
- The third entry is the agent's OSC port for agent communication (will most likely be 7000 for all agents)
- Each line represents one neighbor agent.
- Run MATOS on each computer from the
build
folder./MATO -i 0 -o 7000 -p pd/main.pd -n ../neighbors.txt -s 4
- make sure to change the agent ID for each agent (i.e. the
-i
parameter) - For the Timeline (ie beat clock) to work the last agent started should be designated as the master using the
-m
switch./MATO -i 0 -o 7000 -p pd/main.pd -n ../neighbors.txt -s 4 -m
To test sensor input, I've also include a TouchOSC patch to mimic various inputs. and controls.
- Load the patch into TouchOSC - https://hexler.net/docs/touchosc-configuration-layout-transfer-wifi
- Configure TouchOSC with the IP address and Port of the agent you'd like to communicate with
- Remember the sensor input port is the port entered in the command line parameters + 1000 (so 8000 by default)
- Open the Patch
- You should now be able to start and stop the Agent's sound generation, modify the tempo and volume, and change the agent's state
- NOTE: To see how agents communicate, notice that when you change the tempo on one agent, the other agents will follow.
While any PD patch that meets the MATOS PD API can be used, this iteration of MATOS includes a PD patch
consisting of a 16 bar sequencer for 4 samples. The patch can be found at pd/main.pd
. The pd
folder
also contains two folders for storing your samples and states. States are simply the sequences you designed
and saved through the PD patch.
To edit the patch, first launch pd/main.pd
(you can close the "modules" window that opens automatically).
The patch is preloaded with 4 default samples but you can load new samples by putting PD in edit mode (cmd-e
) and
typing the path to the sample in the message box. Then, back in interact mode (cmd-e
again), click the bang to
load the sample.
To start the sequencer click the orange bang at the top left. Once you've designed a sequence you like, click the green
save button and save the file to the states
folder with the name state#.txt
where # is the number of the state. You
can load saved states with the load button or by entering the state number in the state number object.
(One possible workflow that I've used while designing MATOS, is to build a sound world in Ableton Live. Then decide which samples should go to a given agent, export the samples, load into the PD patch, and edit the sequence.)
All new PD patches should implement the following receive addresses:
[r $0-start-fromCpp]
- receives a bang[r $0-tempo-fromCpp]
- receives a float value[r $0-state-fromCpp]
- receives a float value (representing the state number to load)[r $0-volume-fromCpp]
- receives a float value
Additionally, each receive object should have a corresponding send object that sends the result of the change back to MATOS
[s $0-start-toCpp]
- send a bang[s $0-tempo-toCpp]
- sends a float value[s $0-state-toCpp]
- sends a float value (representing the state number to load)[s $0-volume-toCpp]
- sends a float value
New send and receive objects can be added to the patch but must also be implemented in C++ (ie LibPD) for sending and receiving data between MATOS and PD.
An agent in MATOS encapsulates all components necessary for agent understanding the world, communication between sensors and other
agents, and sound generation. Most of the development of a new Agent happens in the Agent's init()
method. Here you can register
BDI components (see below), add new sensor and agent communication monitors. See Agent.cpp for a demo implementation that handles
sensor and agent communication, sound generation and BDI processing.
Currently, beat alignment is handled by a single agent that sends out tick messages to all neighboring agents. Given a specified tempo, the Timeline sends out 4 ticks per beat. Each tick triggers the PD patch to play the next beat in the sequencer. The Timeline also performs a simple latency calculation before starting to estimate the latency of messages to each neighbor. Ticks are sent in order of latency to ensure ticks are received at all agents at approximately the same time.
NOTE: The Timeline is being designed to allow for decentralization in a future iteration.
MATOS implements a BDI model for rational agents
- See the MATOS Design Document for full details on adding new behaviors and goals
Beliefs represent information about the world that the agent obtains through sensors and other agents. In MATOS, beliefs are composed of an url styled ID indicating the Agent ID and Parameter Name, the parameter name, and a list of parameter values. For example, a belief about agent #5’s tempo would be represented as:
- ID: /tempo/5
- parameterName: tempo
- parameterValues: [90]
The main Agent has a belief database that is a map of type shared_ptr<map<string, shared_ptr<Belief>>>
- NOTE: The ID uses a URL style format which makes it easy to assign from an incoming OSC Message (see Agent Communication Section for more details).
A desire (or Goal) represents a condition that the agent is trying to achieve. In the case of MATO, a condition is a boolean expression that evaluates to either true or false based on the current beliefs of the agent. For example, a goal maybe represented by the condition:
myTempo % worldTempo == 0 or worldTempo % myTempo == 0
- Implemented in code it is a vector of strings (see Agent.cpp)
Each goal has an associated intention, ie action, which represents what the agent should do based on the result of goal evaluation using the agent's belief database.
void tempoAction(bool goalMet, Beliefs beliefs)
{
if (! goalMet)
updateState(“tempo”, beliefs.worldTempo)
}
In MATOS, actions are represented by callback functions that are registered with the goal, see the Agent.cpp for examples.
The interpreter is the reasoning system of the agent; given a set of beliefs, desires and intentions what should the agent do?
For each Intention registered with the agent, the interpreter processes the beliefs to generate parameters for the intentions.
Then Each intention is evaluated to see if the goal is met and an action is generated. The interpreter then selects actions to
perform and executes those actions.
These steps run inside an update loop that is run every N milliseconds (currently set to 500 ms).
Agents must be able to acquire knowledge about the world (including other agents in the world) to generate beliefs about the surrounding environment. General world knowledge is acquired through various types of sensors, such as proximity sensors, light sensors and tangible interfaces to name a few. Agent knowledge is shared and acquired by communicating directly with an agent's neighbors. This information is processed by the BDI model to come up with a set of actions that change the state of the world; in our case actions will generally map to some sound output.
In MATOS, all input is handled though the Open Sound Control(OSC) messaging protocol (with the exception of audio input, although an independent audio processing component could be implemented to send OSC messages for certain auditory events).
MATOS has a monitor system that allows sensor drivers to communicate with the Agent via OSC. This way
sensors and feature extraction are implemented independently of the Agent and updates the belief database by communicating through
the OSC API for sensors. Information from each sensor will update the belief database with the given parameter value.
For example, an OSC Message /volume/1 90
indicates that the belief database should be updated with the volume for agent 1 to
a value of 90.
Using OSC also affords the opportunity to use Wekinator for processing sensor data with machine learning algorithms.
OSC Callbacks are implemented in the Agent.cpp to indicate how certain input from a sensor should be processed.
Agent Communication is also performed through OSC. Each agent is started with a list of neighbors that includes the IP Address and the
OSC port of all neighbors (see the neighbors.txt
file for an example). Anytime an agent’s state is changed, the agent sends a
message with the parameterName and parameterValue to all registered neighbors. The receiving agents then update their belief
database based on the incoming messages. Using OSC allows each agent to independently decide what to do with incoming messages.
OSC Callbacks are implemented in the Agent.cpp to indicate how agent input should be processed.
- NOTE: Each agents’ list of neighbors can be different affording possibilities for some interesting graph based communication patterns.
- Simplify the MATOS BDI API. Possibly by integrating a scripting language such as Python for all behavior, goal and action implementations or by adding a GUI.
- Decentralize the timeline (ie the beat clock)
- Improve the Sound design interface. Given the right resources, it would be interesting to sync Ableton Live directly with the libPD and MATOS.