/Synthe

Bodystorming Human-Robot Interactions

Primary LanguagePython

Synthé

Synthé is atool that allows designers of human-robot interactions to bodystorm their interactions. Bodystorming is an embodied design method that involves brainstorming via role-playing, and in an environment that is true to the product being designed [1].

In bodystorming, a design team acts out an example interaction between a human and a robot, where one user plays the human and another plays the robot. Synthé records the speech and getures of the designers, and employs program synthesis to create an executable human-robot interaction program. This program can be simulated on a Nao robot.

Software and Hardware

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Software Prerequisites and Installation

To install Synthé, the following are required:

  • Python 3, with the following installed:
    • watson_developer_cloud
    • google-cloud-speech
    • rasa_nlu
    • PyAudio
    • sounddevice
    • python bindings for the z3 prover

If PyAudio fails to install, you may need to install libportaudio2 and portaudio19-dev onto your system. On Ubuntu, this can be done with sudo apt-get install libportaudio2 portaudio19-dev.

Rasa-nlu will likely require sklearn_crfsuite and spacy to work. Spacy requires a model of the English language to work, which you can install with python3 -m spacy download en.

You must also have the following installed:

  • D3
  • Dagre D3

Once the above libraries are installed, pull the code from this repository.

git clone https://github.com/Wisc-HCI/Synthe.git

To use D3 and Dagre D3, place the node_modules folder in /learner.

Hardware Prerequisites

Microphones

At minimum, Synthé requires only one microphone be connected to the computer that it is run on. This can be the default microphone on any modern laptop. To acheive full functionality of Synthé, two microphones should be connected. Synthé will automatically detect the presence of microphones.

For your speech to be recognized, you will need to have an account with IBM Watson. Generate username and password credentials with your account, and place them in the intent_parser/ibm_keys.json file.

Gesture Recognition

Synthé currently supports gesture recognition through the Myo Armband [2]. Gesture recognition is not required to run Synthé, though. To support the usage of any device that records acceleration and orientation, Synthé interfaces can interface with these technologies through a server-client setup. Support for other gesture recognition technologies must be developed separately.

If you plan on using the Myo armband, you will need to install the SDK from https://support.getmyo.com/hc/en-us/articles/360018409792-Myo-Connect-SDK-and-firmware-downloads.

Robot Simulation

Synthé also supports the ability to simulate programs created through its program synthesis engine on a real robot. The only robot currently supported is Nao [3]. To support the usage of any robot that simulates programs created through Synthé, Synthé interfaces with these robots through a server-client setup. Support for other robots must be developed separately.

If you plan on simulating the code on a Nao robot, you will need to install the naoqi SDK (version 2.1.4) from softbank robotics. Place the SDK in the /robot folder. Additionally, you will need a credentials file for Google Cloud, and will need to create a GOOGLE_APPLICATION_CREDENTIALS environment variable with the location of this file.

Running Synthé

If this is your first time running Synthé, you will ned to generatean example speech model. The example we use for demonstration involves a robot sitting at a help desk at a store. From the intent_parser directory, execute sh create_speech_data.sh to generate the example model.

Once the model has been generated, from the top directory, execute ./synthe.

Note that without microphones and a robot hooked up to Synthé, capturing designers' intent is limited to the command line. When you press record, you have the option to enter the inputs and outputs of the human and the robot manually on the command line.

Citing Synthé

To cite Synthé, please use the following bibtex:

@inproceedings{porfirio2019bodystorming,
  title={Bodystorming Human-Robot Interactions},
  author={Porfirio, David and Fisher, Evan and Saupp{\'e}, Allison and Albarghouthi, Aws and Mutlu, Bilge},
  booktitle={Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology},
  year={2019}
}

References

[1]: Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003). Understanding contexts by being there: case studies in bodystorming. Personal and ubiquitous computing, 7(2), 125-134.

[2]: The developer blog for myo: https://developerblog.myo.com/

[3]: Pot, E., Monceaux, J., Gelin, R., & Maisonnier, B. (2009, September). Choregraphe: a graphical tool for humanoid robot programming. In RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication (pp. 46-51). IEEE.