Take a picture to make music with the computer.
This is an experiment built with machine learning that lets you make music with the computer just by taking a picture. It uses image recognition to label what it sees, then it turns those labels into lyrics of a song.
https://aiexperiments.withgoogle.com/giorgio-cam
This is not an official Google product.
Built by Eric Rosenbaum, Yotam Mann, and friends at Google Creative Lab using MaryTTS, Tone.js, and Google Cloud Vision API. Check out more at [A.I. Experiments] (https://aiexperiments.withgoogle.com).
The client-side javascript application captures images using WebRTC. When the user hits the shutter button, an image is sent to the server which then returns an array of labels and confidence scores for that image using Cloud Vision. These labels are dropped into a rhyming template to create the next phrase that the computer will speak. To get the audio of that phrase, the client makes another request to the MaryTTS (text to speech) server which returns a wav file of the audio. That audio is then synced to the music using Tone.js.
To build the client-side javascript, first install node and webpack. Then you can install of the dependencies of the project by typing the following in the terminal:
cd static
npm installThen build all of the files
webpack -pThe back-end uses Google App Engine to serve static content and mediate between the two other back-end services: Google Cloud Vision and MaryTTS.
You will need to first enable the API and generate credentials. Under "Key type", use "JSON" and then download the key.json file.
Add your json key to the env_variables section of your .yaml file like so:
env_variables:
GOOGLE_APPLICATION_CREDENTIALS: PATH/TO/CLOUD_VISION_KEY.jsonDownload and install MaryTTS. Then run the MaryTTS Server. Add the IP Address and Port number that MaryTTS is running on to server/mary.py. The default location is http://localhost:59125
MARY_TTS_URL = 'http://127.0.0.1'
MARY_TTS_PORT = '59125'To install the dependencies so they can be launched within Google App Engine they will need to be installed into a local folder (documented here).
cd server
mkdir lib
pip install -t lib -r requirements.txtThen follow instructions on launching your App Engine code.
Copyright 2016 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.