V20 Voicemail Speech-to-Text Google Integration & API Billing

[SethTalley]

• 3CX Version, Version 20.0 Update 0 (Build 1620 Release)
• Server OS, Debian 10, I think
• Is the 3CX Server Hosted and where? 3CX-hosted
• IP Phone Make/Model/Firmware version, n/a
• Provisioning Method: STUN
• Trunk Provider or VoIP Gateway Make/Model, VoIP.ms/Telnyx/Flowroute
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

So in the past I've had our voicemail transcription break whenever I neglect to give Google its 20 cents or whatever. I'm glad I created and populated this thread because (1) Google sucks at reminding me to pay them (2) You can't prepay them (3) every time a credit card number changes I forget about it until too late and transcription stops.

Up through V18 the solution was to regenerate keys because once there's been a billing error the API key is garbaged. V20, however, uses Google's API V2, which uses a .json key rather than an API string.

API V2 is more expensive than V1. I know this because the "budget" I had to create when first installing speech to text in 2018 or whatever woke up two weeks ago and told me I'd used 50% of my budget. Then day before yesterday it told me I'd used 90% of my budget.

Yesterday? Speech to text stopped working about noon. I checked it about 5:30 pm and that number had increased to 92%. I increased my budget from $1/mo to $3/mo wondering if it would fix speech-to-text.

Lo and behold, it's working again this morning.

I'm mostly writing this so that when I'm tearing my hair out over speech-to-text not working (it's one of our most essential business functions so thanks very much for implementing it!) my Googling will find my V20 solution rather than my V18 solution? But also to let everybody else know and invite any tips or improvements to make this situation easier to diagnose and treat.

 

[OlegR_3CX]

Hi there.
Did you try reaching Google support on this matter to see if they can comment on this for that specific time?
Also, once your call ended, you could see your recordings got archived to the Google bucket?

 

[SethTalley]

The last time I reached out to Google they did nothing for two weeks, at which point I'd solved my own problem. My experience with Google has been that they know nothing, do nothing, and care not, so I didn't inquire.

As far as buckets, it does not appear that operation was ever interrupted - the outage was halfway through March 20 and ended early March 21. This is not my expertise, however, so if you can walk me through getting you a better report I'm happy to do it.

 

[OlegR_3CX]

As you know the transcription will take place once the recording/voicemail is uploaded to the Google service, in your case the bucket. So in the event the recording can't get uploaded there, it will re-try. Now, it can also be the case if Google service itself does do what it is supposed (no perfect system out there).
If you had your Hosted instance logging in verbose, and if there were any errors, you could look in the system service log (after you generate the support info file you will find them there), but if not errors, and the recording is uploaded to the bucket (it will be marked with an archived icon under the recording then) then maybe its something with the actual Google service and they could advise.

 

[SethTalley]

3CX is still advising that we run verbose logging sparingly, correct?

The next time this happens I'll put it in verbose long enough to capture a failure before I mess with billing.

 

[OlegR_3CX]

The default is not verbose and you can enable it. Once enabled, just don't restart services as it will reset and revert the settings.

 

[ivank]

Hello Seth!
In V20 we've added informational event 16384 which is recorded whenever transcription status changes.
When transcription stops working, one can check the Event Log for this event which would contain the reason of failure (see attached screenshot). This might provide some insight on the cause even without checking the logs.
On a busy system with many events one better input "16384" into the "Search" field to quickly find these events only.

 

[SethTalley]

Hey, Ivan. So 16384 is not triggering. What *is* triggering is 30052 - for some reason I have exactly one mobile number that is absolutely infuriating 3CX.

not only that but when I go into my page groups and ring groups it's demanding that everything have a department or it won't let me move on. Then it requires a 3CX talk extension.

I upped my "budget" BTW from $1 to $3. Transcription and emails stopped at noon. At 6pm Google sent me an email saying I'd reached 50% of my budget. I've upped it again. I'll bet a dollar transcription works tomorrow.

I have a full set of verbose log files. Shall I send them in?

 

[SweetAction]

IIRC, 3CX v20 can transcribe more then the first minute (which v18 only did first minute), so it could cost more.

 

[SethTalley]

Putting a pin in this: 3CX's email relay was blacklisted by their ISP and describing this as "speech to text fails" rather than "speech to text emails not being received" cost me $75.

Word to the wise: if you don't get a 16384 it's not actually a speech to text failure.

 

[SethTalley]

Adding for posterity: the emails still occasionally take a long time to arrive (3 hours two days ago). Nothing changes on our side. I suspect this is, again, the 3CX email relay encountering shenanigans.

 

[biggreyphone]

All of a sudden my VM emails also started taking 2.5-3 hours to arrive on google translate. I switched to AI Whisper and this seems to resolve it.

 

[SethTalley]

All of a sudden my VM emails also started taking 2.5-3 hours to arrive on google translate. I switched to AI Whisper and this seems to resolve it.

Could you outline the process as a favor to me? I'm good at following directions but I'm no coder and my ability to screw this up on my own is absolutely unparalleled.

 

[biggreyphone]

Follow this thread.

https://www.3cx.com/docs/manual/openai-transcription/

 

[biggreyphone]

You can easily switch back and forth between Google and Whisper as needed so I wouldn't hesitate to try out both.