Both sides of conversation not heard - but both sides recorded.

[em97jch]

3CX Version: Professional Annual 18.0 (Build 461)
Server OS: Debian
Is the 3CX Server Hosted and where? Data Centre (VPN)
IP Phone Make/Model/Firmware version Polycom 5.6.0.17325
Provisioning Method: VPN
Trunk Provider: Voiceflex

We are experiencing a strange inconsistent problem with a small number of calls. The call connects (typically from a queue) and both sides start speaking. The conversation can continue for any length of time. Then suddenly neither side can hear each other. The call recording has the complete conversation which typically concludes with both sides are saying "Hello" - "Hello, can you hear me?".

Other calls that are taking place at this time are unaffected and this happens on other extensions and when a call is transferred from other queues. The calls can be queued for a number of minutes.

The only pattern that we can see is that the problem happens 'around' 16 minutes of total duration (including queuing). The problem calls that have been queued for longer than 16 minutes cannot be heard at either end as soon as they are picked up.

However, there are plenty of successful calls with durations of 16 minutes and significantly longer. So it's fair to say that all the problem calls have lasted at least 16 minutes, but not all calls that last 16 minutes or longer are problem calls!

Any suggestions?

 

[ChrisC_3CX]

To first get a bit more information on this:

IP Phone Make/Model/Firmware version Polycom 5.6.0.17325

What exact model are you using? Provided it is one of the 3CX Supported devices it seems you may be running old firmware so this would be one of the first things I recommend sorting.

Provisioning Method: VPN

Just to clarify, we have 3 provisioning methods Local, SBC and STUN. Do you mean they are provisioned using the LOCAL LAN provisioning method but are actually physically remote using a VPN to accommodate for this?

Then suddenly neither side can hear each other. The call recording has the complete conversation which typically concludes with both sides are saying "Hello" - "Hello, can you hear me?".

Have you confirmed with both call participants that they could not hear anything once the issue occurred? The reason I'm asking you to confirm is because it would indeed be really strange for this to be the case while having audio in the call being recorded. It would essentially mean that you have audio coming in from you provider reaching the PBX but never reaching the IP Phone and at the same time, have audio from the IP Phone go through the PBX but never reach the caller end, having both these effects sharing the same cause would be strange so confirming what both callers hear during the actual call is vital.

 

[em97jch]

Thanks for the reply, ChrisC_3CX.

The Polycom model is the VVX 400 - which I appreciate is EoL. Has been used for several years - although worth mentioning that the call load has increased recently (hence the increase in duration). The 3CX server is VPN'd to the network, so it's local.

Apologies. I was wrong. The receiver can still hear the call - the external caller can not hear anything, but both sides of the conversation are recorded.

 

[ChrisC_3CX]

Apologies. I was wrong. The receiver can still hear the call - the external caller can not hear anything, but both sides of the conversation are recorded.

Excellent, that means that incoming RTP traffic (media traffic) is reaching your PBX and then reaching your IP Phone. Regarding outbound RTP traffic, it seems that, the IP Phone is sending it and evidently is reaching the PBX (since it's in the recording) but not reaching the external caller. Most probably the only thing you are in control of that's still in the equation is your network's firewall so there is still a chance it is something on your end but reaching out to the SIP Provider to have them confirm if they are receiving the RTP traffic or not might be the next best thing to do.

If you want to confirm everything is ok on your end you might want to check if you have any outbound restrictions on your firewall. You could also look into running a packet capture on the firewall's WAN and LAN interfaces (if possible) while you're on a problematic call. You could then compare the RTP traffic in the two packet captures to see if there's anything off.

 

[em97jch]

Hi Chris,

Getting a full packet trace of the problematic RTP is a needle in a haystack as there doesn't appear to any pattern as to when it will occur. The buffer gets full within seconds on a generic trace. Tracing 5060 gives some details - but does not isolate the RTP problem.

The SIP provider has sent through a graph of the RTP traffic - "...Nothing seems to be peculiar in the RTP streams and it looks like they end at that the same time..."

I've also tried changing the "Transport Protocol" to TCP (https://www.3cx.com/community/threads/trunk-options-transport-protocol-gets-reset.69994/) - same/similar symptoms.

Firewall is a SonicWall NSa 3600 and passes the check. (We have other 3CX installations behind this firewall, but this particular customer does have long queues and a lot of calls....) "Enable consistent NAT" is checked and "SIP Transformations" are disabled.

Unfortunately, I suspect we may lose this customer......I may suggest moving them to 3CX hosting as a last gasp effort to keep them.

Anything else we can try?

 

[JohnS_3CX]

Hello,

Firewall is a SonicWall NSa 3600 and passes the check.

Do keep in mind that the checker does not inspect ALL the RTP ports, just the first few and last few.

So, as the system makes more calls, and the ports keep incrementing, you may be hitting some blocked ports.

Double-check the UDP range 9000 to 10999 and ensure they are all actually forwarded just to be on the safe side.





The ultimate confirmation would be to run tshark on the server, dumping the capture on your drive until the first incident occurs.

Hangup, stop the capture and inspect it. You need to confirm that the outbound RTP traffic has actually left the 3CX NIC before you reach any conclusions.

Example, feel free to adjust:

apt-get install tshark
mkdir pcap
chmod +7777 pcap
tshark -i any -b filesize:10240 -w pcap/capture.pcap

 

[em97jch]

Thanks John

I've just this hour got a full pcap of a call that cuts out - which is an achievement in itself!

Audio was lost about 40 seconds from the end when the operator hung up. The RTP traffic seems uninterrupted all the way.........BUT up to 40 seconds from the end PCMU was used all the way. Voiceflex then changes to PCMA and the RTP traffic continues both ways but now PCMA one way and PCMU the other way.

I don't know too much about codec's, but the timing seems too coincidental. Can anyone confirm that this would result in one way audio?

I've disabled PCMA on the trunk and will continue monitoring.

 

[JohnS_3CX]

The codec usually changes when there is an event that triggers it (ie. hold, unhold, transfer).

I don't think the provider would simply start sending a different codec without signalling 3CX first.

Otherwise the receiving ends would not know how to decode the incoming stream, because they would be unaware of the change and at least one side could end up hearing silence (or noise) as the decoder fails.

 

[em97jch]

and at least one side could end up hearing silence (or noise) as the decoder fails.

One side hearing silence, is what we are experiencing.

The codec usually changes when there is an event that triggers it (ie. hold, unhold, transfer).

Yes, calls experience the issue after being put on hold/unhold or transferred out of the queue.
Additionally, in the pcap captured call we have it cutting out (and swapping over from PCMU to PCMA) right in the middle of the conversation.

The provider has responded with "...I am unsure why the codec would change unless this was negotiated earlier by the PBX. You have done the correct thing by disabling the codec..."

It looks like disabling PCMA on the trunk has resolved the problem.
Is there anything in 3CX that could account for this?

 

[NickD_3CX]

It looks like disabling PCMA on the trunk has resolved the problem.
Is there anything in 3CX that could account for this?

I'm glad you found a way around this.

The only thing I could think is if you had enabled "Supports Re-Invite" in the SIP Trunk "Options" tab, but that would mean that someone has tinkered with the settings, because that is always off by default.
If that was on, and after a hold/unhold attempt, this *could* happen, but other than this scenario I can think of something else off the top of my head.