Has anyone seen this? Gemini Speech Sensitivity?

UCMUserAZ

Customer
Joined
Dec 19, 2023
Messages
159
Reaction score
248
I have been testing a few gemini call scripts with 3cx this week. Very powerful and awesome I have to say. It seems the speech sensitivity once connected with Gemini is really touchy. Testing without speakerphone it works really good. When on speakerphone (iOS latest; 17pro) Gemini can interrupt sometimes. I tested with just a simple greeting and it sometimes repeats. If I put speakerphone on with the mic muted, it works 100%. Not sure if the iphone mic is picking up the speakerphone audio or what or an iphone issue. I don't think this is anything on the 3cx side, but just how Gemini is listening. I see gemini has some realtime_input_config settings around: https://aistudio.google.com/docs/live-api/capabilities
  • activityHandling (NO_INTERRUPTION)
  • automaticActivityDetection.startOfSpeechSensitivity
  • automaticActivityDetection.endOfSpeechSensitivity
  • automaticActivityDetection.silenceDurationMs
I do not think 3cx exposes these settings through StartGeminiVoiceSessionAsync() or global parameters. Do you think it would be possible to add support for passing realtimeInputConfig when initializing the Gemini Live session? Only reason I ask, I know some of our callers use speakerphone and we would hear this feedback from them on the Ai voice agent. Not sure if Grok or OpenAi also have this kind of sensitivity.

If anyone out there has a gemini call script, I would be curious if speakerphone produces the same? It is just odd where if I mute the mic on speakerphone it works perfect so would not think anything in my prompt.

Thanks in advance!
 
The mute test is a good clue.

Please clarify the call route; is the iOS making an incoming call through the PSTN network? Is it caling as an extension? Can we have an explicit description of the call route end-to-end?

Try the same greeting at low and high speaker volume, then repeat it on a second (different model, ideally not iOS) phone using the same call route and Gemini model.

If the interruptions get more frequent as the volume goes up, that would point toward speaker audio feeding back into the mic.

If different model phones give different experiences, that adds hints towards noise cancellation settings.
 
  • Like
Reactions: KyriacosS_3CX
The mute test is a good clue.

Please clarify the call route; is the iOS making an incoming call through the PSTN network? Is it caling as an extension? Can we have an explicit description of the call route end-to-end?

Try the same greeting at low and high speaker volume, then repeat it on a second (different model, ideally not iOS) phone using the same call route and Gemini model.

If the interruptions get more frequent as the volume goes up, that would point toward speaker audio feeding back into the mic.

If different model phones give different experiences, that adds hints towards noise cancellation settings.
Call Flow: on iphone, Call DID, comes thru our SIP trunk at teleco, enable speakerphone, call hits SBC, 3cx runs call script from that DID, gemini live plays first greeting, then will repeat it. Almost like it heard itself. I just tested the base callscript with no mods and it repeats when on Speaker. Mic muted works fine and without speakerphone works fine. I will test some other phones. Thanks.
 
Some additional testing, same on Samsung speaker. Also, not sure if intended. Does barge in work for others testing these scripts? Like if the Ai agent is talking and you interrupt it. I am finding it wants to finish what it is saying before allowing you to interrupt if it is answering a question. Not sure if others have seen that. Thanks.
 

Latest Posts

Forum statistics

Threads
112,551
Messages
593,186
Members
165,762
Latest member
LchaplinTCG