Session configuration not set according to parameters

[arnaudbreton]

Hi,

I'm facing a weird bug where the following code doesn't lead to the session configuration to being properly sent.

On the browser-side (React Native through Expo):

  client.updateSession({
    turn_detection: { type: 'server_vad', threshold: 0.5 },
    input_audio_transcription: { model: 'whisper-1' },
  });

await client.connect();

On the relay server:

clientWs.on('message', (data: any) => {
        const event = JSON.parse(data);
        console.debug(JSON.stringify(event, null, 2));
        console.log(`Relaying "${event.type}" to OpenAI`);
        openAIWs.realtime.send(event.type, event);
}

Most of the time I get the following logs:

Relaying "session.created" to Client
Relaying "session.updated" to Client

In rare occasions, or if I set a breakdown in the updateSession method, I get the following log:

Relaying "session.update" to OpenAI
æ
  "event_id": "evt_ZEcRWpyaoMgCY4amY",
  "type": "session.update",
  "session": æ
    "modalities": Æ
      "text",
      "audio"
    Å,
    "instructions": "",
    "voice": "alloy",
    "input_audio_format": "pcm16",
    "output_audio_format": "pcm16",
    "input_audio_transcription": æ
      "model": "whisper-1"
    å,
    "turn_detection": æ
      "type": "server_vad",
      "threshold": 0.5
    å,
    "tools": ÆÅ,
    "tool_choice": "auto",
    "temperature": 0.8,
    "max_response_output_tokens": 4096
  å
å

This prevents server_vad to be effective as the session configuration isn't sent.

Thanks for your help

[arnaudbreton]

As a temporary fix, I'm sending the session configuration directly and once the app has received the session.created event.

switch (event?.type) {
            case 'session.created':
                openAIWs.realtime.send('session.update', config);
                break;
}