Google Assistant API Sample for Android Things

This sample shows how to call the Google Assistant Service from Android Things using gRPC.

It records a spoken request from the connected microphones, sends it to the Google Assistant API and plays back the Assistant's spoken response on the connected speaker.

Pre-requisites

Android Studio 2.2+.
Android Things compatible board.
AIY Projects Voice Kit or supported microphone and speaker (See audio configuration.
Google API Console Project.

Run the sample

Configure the Google API Console Project to use the Google Assistant API.
Download client_secret_NNNN.json (type: Other) from the credentials section of the Console.
Install the google-oauthlib-tool in a Python 3 virtual environment:

python3 -m venv env
env/bin/python -m pip install --upgrade pip setuptools
env/bin/pip install --upgrade google-auth-oauthlib[tool]

Use the google-oauthlib-tool to generate credentials:

env/bin/google-oauthlib-tool --client-secrets client_secret_NNNN.json \
                             --credentials app/src/main/res/raw/credentials.json \
                             --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
                             --save

Make sure to set the Activity Controls for the Google Account using the application.
On the first install, grant the sample required permissions for audio and internet access:

./gradlew assembleDebug
adb install -g app/build/outputs/apk/app-debug.apk

On Android Studio, click on the "Run" button or on the command line, type:

./gradlew installDebug
adb shell am start com.example.androidthings.assistant/.AssistantActivity

Try the assistant demo:
- Press the button: recording starts.
- Ask a question in the microphone. After your question is finished, recording will end.
- The Google Assistant answer should playback on the speaker.

Audio Configuration

By default the sample routes audio to the I2S Voice Hat on Raspberry Pi 3 and default audio on other boards (on-board Line out or HDMI/USB if connected).

You can change those mappings by changing the USE_VOICEHAT_I2S_DAC constant or replacing the audio configuration in the onCreate method of AssistantActivity with one of the following:

// Force using on-board Line out:
audioInputDevice = findAudioDevice(AudioManager.GET_DEVICES_INPUTS, AudioDeviceInfo.TYPE_BUILTIN_MIC);
audioOutputDevice = findAudioDevice(AudioManager.GET_DEVICES_OUTPUTS, AudioDeviceInfo.TYPE_BUILTIN_SPEAKER);

// Force using USB:
audioInputDevice = findAudioDevice(AudioManager.GET_DEVICES_INPUTS, AudioDeviceInfo.TYPE_USB_DEVICE);
audioOutputDevice = findAudioDevice(AudioManager.GET_DEVICES_OUTPUTS, AudioDeviceInfo.TYPE_USB_DEVICE);

// Force using I2S:
audioInputDevice = findAudioDevice(AudioManager.GET_DEVICES_INPUTS, AudioDeviceInfo.TYPE_BUS);
audioOutputDevice = findAudioDevice(AudioManager.GET_DEVICES_OUTPUTS, AudioDeviceInfo.TYPE_BUS);

Device Actions

With Device Actions, you can control hardware connected to your device. In this sample, you can turn on and off the LED attached to your Android Things board.

Follow the guide here to learn how to register your device.

After you register your device model and id, replace the device model and instance PLACEHOLDER values in AssistantActivity:

        private static final String DEVICE_MODEL_ID = "my-device-model-id";
        private static final String DEVICE_INSTANCE_ID = "my-device-instance-id";

Handle a Device Actions response if you get one.

        mEmbeddedAssistant = new EmbeddedAssistant.Builder()
                ...
                .setConversationCallback(new ConversationCallback() {
                    ...
                    @Override
                    public void onDeviceAction(String intentName, JSONObject parameters) {
                        // Check the type of command
                        if (intentName.equals("action.devices.commands.OnOff")) {
                            try {
                                boolean turnOn = parameters.getBoolean("on");
                                mLed.setValue(turnOn);
                            } catch (JSONException e) {
                                Log.e(TAG, "Cannot get value of command", e);
                            } catch (IOException e) {
                                Log.e(TAG, "Cannot set value of LED", e);
                            }
                        }
                    }
                }
                ...
        ...

Try it:

"Turn on"
"Turn off"

The LED should change states based on your command.

License

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0