Audio drops on outbound calls after 10-20 seconds

[contoured]

One of our customers has an issue where outbound calls are frequently dropping audio after roughly 10-20 seconds of conversation. When this occurs, neither side of the conversation can hear anything, but the call stays active.

There are 5 remote offices connected to this PBX via SBC. 4 of the 5 SBCs are Raspberry Pis and the 5th is a Windows 10 Pro Intel NUC. The only office that appears to be having issues is on the Windows SBC.

Background: I updated the PBX and SBC to the latest v16 update 3 and that appears to be when the problems first started. The PBX and SBC had been running well for over a year before the upgrade. One thing I should mention is that when I performed the upgrades, I stepped the system up from v15.5 to v16 update 3 in a single night.

Things I have tried:

• SBC - Restarting SBC Service
• SBC - Rebooting SBC
• SBC - Uninstall SBC then Reinstall
• SBC – Change Security to TCP
• SBC – Force 5090 traffic through of a secondary WAN interface
• SIP Trunk – Enable “Supports Re-Invite” and “Support Replaces”
• SIP Trunk – Transport protocol UDP
• PBX - Restarting Services

I was able to packet capture an example of the issue. The first thing I noticed in the call flow was that the initial SIP INVITE from 3CX goes to 64.2.142.107 (outbound.vitelity.net), then the 100 Trying, 180 Ringing, 183 Session Progress, and 200 OK are all returned from that same IP. However, 3CX responds with 200 ACK to a different IP 64.2.142.108 (also outbound.vitelity.net). Then the call seems to fall apart with the .107 IP sending more 200 OKs to the PBX, and the PBX sending more ACKs to the .108 IP.

I contacted Vitelity support and they confirmed what I saw and that after review said they “don’t see any cause to us directing you to 64.2.142.108. That the contact header shows to send responses back to the 107 IP address”.

Any insight on how to attack this problem is greatly appreciated!

The system is configured in a fully supported manner. Here is the system information.

• 3CX Version, e.g. Professional Perpetual 16.0.3.676
• Server OS, e.g. Debian 9
• Is the 3CX Server Hosted and where? Yes/ Google
• IP Phone Make/Model/Firmware: Yealink T46S 66.84.0.35
• Provisioning Method: Local / VPN / STUN / SBC: SBC
• Trunk Provider or Gateway Make/Model: Vitelity – With default template
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

 

[AWS2P]

Do you have more than one trunk from Vitelity on same PBX ?

is or Are trunks set with IP 64.2.142.107 or domain name?

 

[contoured]

Do you have more than one trunk from Vitelity on same PBX ?

Yes I have a unique trunk for each office for Caller ID and E911 purposes.

 

[AWS2P]

Are your trunks IP based register or Account based?

 

[contoured]

Are your trunks IP based register or Account based?

They are IP based.

 

[AWS2P]

So may be problem is here, in trunk settings, similar case already happens with IP based trunks coming from same provider. I don't remember what need to be set in trunks to fix this, try to check this in inbound parameters in trunks

 

[AWS2P]

i'm going to sleep , sorry i'm tired

 

[JohnS_3CX]

Hi @contoured

Can you run a capture and replicate the failed scenario please?

 

[contoured]

Hi @contoured

Can you run a capture and replicate the failed scenario please?

Hello John,

Yes I can. The issue is intermittent so it takes a few tries but I did successfully capture one such call just a few minutes ago.

On the advice of Vitelity, I set the outbound proxy statically to one of their IP addresses and chose 64.2.142.111. Looking at the call flow, the call starts as expected but then 3CX starts sending ACK 200 to a different IP (64.2.142.108). Listening to the Audio, it gets cut off after roughly 15 seconds.

 

[JohnS_3CX]

Excellent, this is a good capture, please go ahead and open a ticket with support and they will be able to assist you on this exact issue.

 

[swhite]

I've experienced similar that ended up being a UDP timeout setting on local network topology, especially if behind NAT. Vitelity media signalling isn't going to always come from the one server you set for SIP signaling.

 

[BrenttG]

I've experienced similar that ended up being a UDP timeout setting on local network topology, especially if behind NAT. Vitelity media signalling isn't going to always come from the one server you set for SIP signaling.

I think you nailed it right on the head there. UDP timeout is kinda what it sounds like to me too. But where it is happening is the devil in the details.

Does this Issue only affect 1 office, and accordingly, 1 trunk?
|-if so does the issue vanish if you temporarily set that offices outbound rule to use one of the other trunks.
|-|-if so then the problem lives in the trunk.
|-|-if NOT, then the issue is towards that site, and possible the firewall and/or the sbc.

 

[HERETIC]

Following this... I have a similar issue w/ SP3 and calls dropping after ~30s AND IVR conferencing not working – but if I restart the IVR service(s) everything kicks back to normal..

 

[JohnS_3CX]

@HERETIC

You have to be a lot more specific, maybe we can help with the issue. Provide as much detail as possible https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[contoured]

Hello Everyone,

I submitted my packet capture to 3CX support and the confirmed that I'm experiencing a known bug that should be fixed in the next update. In the meantime someone from their support team can apply a hot fix but they will need SSH and portal access. I hope this get resolved for everyone soon. Major issue!

 

[contoured]

The 3CX support technician implemented the hot-fix. It's only been about 4 hours but so far no complaints.

 

[Zol]

What exactly does the hotfix do?

 

[contoured]

What exactly does the hotfix do?

I don't actually know! It seems to have worked for which my customer is grateful.