Sporadic complete audio failure affecting entire department - RTP routing issue on 3CX Cloud instance

[Dre1902]

Environment:

• 3CX Version: 20.0 Update 10
• Hosting: Hetzner
• SIP Trunk: Deutsche Telekom CompanyFlex
• Clients: 3CXPhone for Windows (latest Version)
• Affected users: All users in "Innendienst" department

Problem Description:
All users in our Innendienst department experience sporadic but complete audio failure during calls. When the issue occurs, neither party can hear anything. The problem affects the entire call from the moment it is established and does not recover mid-call. After hanging up, the next call may work normally.


Key findings:

• Problem occurs on both internal and external calls
• Affects all users in the department regardless of headset or PC hardware (tested 3 different headsets, multiple PCs)
• 3CX audio quality log shows RoundTripTime of 2,315,878ms on affected calls
• Quality reports show 0 packets received from the trunk side on affected calls, while the client side sends normally
• Call mode is always Transcoding (PCMA on trunk side vs OPUS on client side)
• Wireshark capture confirms network traffic is clean - packets flow normally between local network and 3CX Cloud, no packet loss on network level
• "PBX delivers audio" is enabled for affected users but does not resolve the issue

What has been ruled out:

• Hardware (3 different headsets, multiple PCs tested)
• Network/Jitter (clean according to 3CX logs and Wireshark)
• Audio device selection (all other audio devices disabled)
• Codecs (multiple combinations tested)
• Local network routing (Wireshark confirms clean traffic to/from 49.12.191.59)
• SIP Trunk (issue occurs on internal calls too)
• Firewall/QoS (configured and verified on WatchGuard M290 and Aruba 2930F)

Suspected cause:
RTP routing or transcoding issue within the 3CX Cloud instance itself. The problem appears to be internal to the PBX as network-level traffic is confirmed clean.


Request:
Please analyze server-side logs for affected calls, specifically RTP stream routing and transcoding processes for Innendienst department users.

 

[NikosT_3CX]

@Dre1902 as I can see your PBX is not hosted by 3CX, its self managed so we do not have access to your machine.
I would recommed telling your 3CX Partner to open a ticket to our Support Portal and then we can take it from there.

 

[Alphabetic]

Check your server hardware and if there is a backup scheduled to run.

Id be more suspicious of the VPS than the installation.

 

[Claas Felix Beyersdorf]

Does this happen to members of other groups also?
Does this happen for all members at the same time (one is having the problem do others have problems at the same time?

Do you have some Edge Protection on the clients that might cause problems?

 

[Dre1902]

@NikosT_3CX I've already tried to troubleshoot this issue together with our 3CX partner. They also reached out to 3CX directly, but unfortunately nobody has been able to give us an answer or even a starting point for troubleshooting so far.

@Alphabetic The server hardware is more than sufficient, and resource usage never gets anywhere near a level where throttling would kick in. Backups don't run during business hours either. They're only scheduled at night.

@Claas Felix Beyersdorf So far, we've only noticed the issue with members of the "Innendienst" (internal sales) group and not with any other group. It can affect several members of the group at the same time, but that isn't always the case. One employee is affected the most, while the others experience it occasionally. We've already checked the edge protection and added all the necessary exceptions. We also disabled it completely for a test run, and that didn't make a difference.

​

 

[NikosT_3CX]

@Dre1902 I can not see any recent ticket in Tech Support for your license key.
The last one was 8 months ago, this is the one you refer to ?
On this ticket we got access to your PBX, we could not replicate the issue you had and then we mentioned to change the codecs and also use the Call Quality Monitoring tool.
After this, there was no response back.

 

[Alphabetic]

What is the network topology where this department lives?

Are there any other extensions on this network?

you need to narrow down your troubleshooting - you've identified the problem occurs with internal calls, so forget your trunks.

You say it happens on 3CX Softphone - what other software do you have on the machines?

Can anyone provision the mobile app or stick in a desk phone temporarily?

 

[Dre1902]

@NikosT_3CX That was most likely a different issue. Back then we had a delay while speaking, but we traced that back to the VPN software and replaced it with a different one. The last contact I'm aware of was last week. Unfortunately, I don't know why there's no ticket for it or how that contact took place.


@Alphabetic The topology is fairly simple. The "Innendienst" (internal sales) department is currently on the same VLAN as most other employees. Employees from another department are connected to the same switch, and everyone is connected via LAN. There's no other site involved, and no VPN is active. The softphone runs locally on the notebooks. The employees mainly work on a terminal server, and only the phone calls run through the notebook. Software such as the Office applications and the VPN client is also installed on the notebooks. All drivers are on the latest version released by the manufacturer. We haven't had the issue with the mobile app yet, but it only occurs sporadically anyway. Unfortunately, I can't test with a desk phone because we no longer use any.