Hardware for local AI

[GWinchester]

I am looking forward to be able to run the AI/transcription locally. however there still seems to be limited information on hardware.
in the transcription engine server instructions https://www.3cx.com/docs/transcription-engine-server/ wich by the way i can only find a link to in one of the bogs (not in the help menu yet) it has GPU: Minimum Recommended (Fast): Nvidia 24GB RTX GPU. in theory an L4 would work?
we have dell PowerEdge R660 so a 5090 just isn't going to fit.
what have people tried and how did it go?

Another suggestion in one of the bogs was a GB10 based unit, however there seems to be no instructions for this type of setup.
Is there any more information available for GB10 or that option is not going to be supported?

 

[nikolascx]

Hi @nikolascx & @Kevin Attard Compagno,
One last question regarding the 3CX AI Server, does it also allow to run the AI agents, especially the AI receptionist ?
If not, is it perhaps planned ?
Would allow to avoid relying on external AI services...
Thank you very much !

Hi @rlg
Why dont you use the programmable extension?
https://www.3cx.com/docs/programmable-extensions/

Clone the github repo, configure the config file,
yarn install
yarn start: openai or yarn start:gemini

Let me know how it goes. We are going to release some videos shortly.

 

[rlg]

Hi @nikolascx,

Thank you for your proposal, looks really promising !
I was however rather thinking about moving speech recognition, reasoning and speech generation to the 3CX AI Server directly.
So that this could be done locally, without having to rely on a cloud AI service such as OpenAI.
Still thinking about customers' data / sovereignty, availability / uptime, costs control etc...

Thank you again !

 

[NikosT_3CX]

@rlg for costs control, please take a look on this blog : https://www.3cx.com/blog/docs/cloud-vs-local-ai-server/

 

[rlg]

@NikosT_3CX thank you for this, I already planned a local 3CX AI Server for v20u10, for transcriptions.
I'm now talking about the other AI part, agents (speech recognition, reasoning and speech generation), which is currently handled by 3CX through OpenAI.
I then share the idea to have it handled by the 3CX AI Server, without knowing if it's feasible or not (@Kevin Attard Compagno gave some hints in his previous post).

 

[nikolascx]

Hi @nikolascx,

Thank you for your proposal, looks really promising !
I was however rather thinking about moving speech recognition, reasoning and speech generation to the 3CX AI Server directly.
So that this could be done locally, without having to rely on a cloud AI service such as OpenAI.
Still thinking about customers' data / sovereignty, availability / uptime, costs control etc...

Thank you again !

In a couple of years time, yes definitely.
But currently, the world is not ready for this.

To do this, there should be a local realtime model (does not exist... yet)
And with a pipeline, you will need GPU Power. So each 3CX you support, needs to get a GPU. How realistic is that?

To put it in perspective, if you have a 4090, plus all the work required to do the Audio pipeline, plus a local model, then you could only make 4-6 sim calls.

Let's ride the wave for now. We are watching attentively, btw what you mention is actually our second goal.

On the bright side, our primary year old feature goal is coming very very soon ... God willing