Dead air on call for 20-30 seconds after volume is ~450 calls in system

[jmatano]

We are running into a issue where it seems like the system is not fully engaging the call until around 20-30 seconds when volume gets to around 400-450 calls in the system. One indicator when this occurs is through the browser extension or softphone client, the screen turns to the in call options, but the button for keypad is greyed out. Once the keypad button becomes functional, audio begins being delivered.

System resources are around 40-50% CPU and memory usage on a VM with 32CPU assigned and 32GB of RAM with SSD storage in Raid 1. It never spikes higher than 55% usage.

One thing to note, when volume reaches that level there is 150+ calls in queue, which include call backs.

We can fix the issue by removing some calls in the system, including QCB's only, which seems odd since they are essentially a placeholder until the call dials out.

I've tried various combinations of SIP providers, and it is not a capacity issue on the provider side because the same cases occur with similar volume regardless of the provider combination.

Any help would be greatly appreciated.

 

[SteveITS]

https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/
OS? 3CX version?

 

[jmatano]

https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/
OS? 3CX version?

3CX Latest Version: 16.0.5.612
Server 2019. Sorry forgot to provide.

 

[cobaltit]

Is the firewall checker passing? If this is an older install the RTP port range was changed some time ago and it's possible your firewall was never updated to reflect that change. The old range used to be 9000-9500 and now it's 9000-10999

 

[jmatano]

Is the firewall checker passing? If this is an older install the RTP port range was changed some time ago and it's possible your firewall was never updated to reflect that change. The old range used to be 9000-9500 and now it's 9000-10999

Yes. One of the steps I should have mentioned I did was I created a new virtual server with a fresh server 2019 installation, and restored 3CX from backup this past Wednesday. I ran the firewall check as soon as the system came back up and was successful. The new port range was updated a while ago. Firewall rule has port 9000-9999 and 10000-14000 open (1028 SC system)

 

[jmatano]

Throwing this out there. The host that this server is on has 32 available threads to it, and I don't have it restricted. During the busiest of times I only see the CPU usage to be around 50-60%. Is there a chance that there is a bottleneck somewhere else in CPU that could be causing the call quality. For example if a couple threads are at 100% but the overall CPU is not nearly maxed out. That clock speed becomes an issue for the used threads?

 

[SteveITS]

if a couple threads are at 100% but the overall CPU is not nearly maxed out

That's what I was wondering...put another way with 8 cores Windows will show a flat 12.5% usage if one core is maxed out and nothing else is going on.

FWIW, Update 6 promises lower resource usage: https://www.3cx.com/blog/releases/v16-roadmap-updates/

 

[jmatano]

That's what I was wondering...put another way with 8 cores Windows will show a flat 12.5% usage if one core is maxed out and nothing else is going on.

FWIW, Update 6 promises lower resource usage: https://www.3cx.com/blog/releases/v16-roadmap-updates/

At this point we are dying around the 500 sim call / ~200 being queue calls. We are looking to purchase a new server simply to throw more processing at it, at hopes that something isn't optimized for multithreading. Does anyone have any insight other than cores, threads, and clock speed, what factors should be used to determine the proper CPU. Looking at possibly 2x Intel 6252 Gold CPU, but it only has 36mb of L3 cache, where that can be improved on other CPU's. Is that a important spec?

 

[SteveITS]

I think I'd open a ticket with support and let them look it over to ensure you're CPU bound.