Poor call quality, choppy, massive delays

[Patrick McNamara]

I've been using 3CX for about 2 months and my system seemed stable and clear, but over the past couple of weeks my call quality has been terrible. Local extension to extension calls are still crystal clear but incoming and outgoing calls are horrible. I've been taking numbers for inbound callers and calling them back on my mobile, not the way my company needs to do business. It's not a bandwidth issue because there is very little running over my network, and it doesn't seem to be related to a specific time. Flowroute is my SIP provider and they have been working on the issue for several days without a resolution. I want to be able to install, configure and resell 3CX phone systems but if I can't make it work for myself I could never have the confidence to resell the product. Any suggestions would be greatly appreciated.

 

[Flash99]

If you want to resell the product you need to be able to understand how it works and the basics of VOIP before you start blaming the product

Do you use real phones or soft phones ? If its phones what codec they use? What codec your SIP provider use ? Where is your server is it on a local network or in the cloud ? Physical server or virtual ?

 

[Patrick McNamara]

I apologize if you think I was blaming 3CX for the call quality issues, because I'm not. 3CX has been an excellent product and I'm looking forward to learning more about VOIP and 3CX. Answers to your questions:

Grandstream GXP2110 Phones on local subnet
On-Premise hosted on my VMWare server running a virtualized instance of Windows7 and 3CX.
I've tried multiple codecs, the default that installs with 3CX and I've tried G722 (which I saw on another post).
I've got an email in to flowroute about their default codec.

This system has been working perfectly for almost 2 months, great call quality, no issues. The problems are recent and local internal calls are crystal clear so I'm convinced it's an external problem. Flowroute keeps saying they are trying to get it resolved via one of their "peers" but nothing has changed and my phone system is basically unusable at this point. Thanks for the help

 

[Patrick McNamara]

Flowroute is using "either G711 or G729. Most customers seem to have better success with G711"

 

[nb]

You have to investigate to see what the problem is and you need to use the concept of elimination.
For sure if you are scenting a network issue (because you complain of external calls only) using G722 is like putting petrol on fire.
Now you need to eliminate flowroute. If I were you I would do the following:
Get a remote phone somewhere (even outside of your country) and connect the phone to the pbx. So then you can compare an external call direct VS an external call via the provider. Then get a friend abroad even in another continent - tell that friend to go to the app store or android play store and download our client, send the friend a welcome email and call each other. Is the quality of an external call ok? Yes no?

If yes, then you can go to the provider and at least you can tell the provider that not all external calls are of bad quality - but those through the provider.

Another perfect elimination example is if you get another provider and test with that. Like a demo account that can call free numbers. Does that give you problems too?

The argument that in the last 2 months everything was ok and now it is not, is not valid in this business. Actually it is not really a valid argument anywhere in troubleshooting. It just shows one thing - that for 2 months everything was good so SOMETHING must have changed. If you kept everything constant (no updates, no network changes, no new devices etc) then yes it could be something in the external environment that changed (like for example overload on your proxies which is something not in your control)

 

[Patrick McNamara]

Just tried another call.

From an employee's mobile phone using the 3CX client over ATT network from Downtown (about 20 miles away)
Call was from the remote employees 3CX client on her mobile phone to a LAN extension, crystal clear.

Should I try another provider and see if the problem goes away?

 

[nb]

Well done...
Well now you have eliminated the fact that your public external interface is healthy. Otherwise a remote extension will get problems.
(The 20 miles away doesn't make much of a difference because that is peanuts for ATT.) but good try.

Try another provider now so you simulate a more realistic test. Then you can get 2 captures from 3CX -> provider - one with bad quality and one with better quality and send it to the provider so they can explain.

 

[Sopock]

I've been taking numbers for inbound callers and calling them back on my mobile, not the way my company needs to do business.

It is always good to have this on mind:
If your company can tolerate an occasional voice quality issue, then you may want to consider having your SIP trunks delivered over your public Internet connection. If not, you just need a dedicated SIP circuit (i.e. fiber, T1, etc.), delivered by your SIP provider. Doing so will guarantee your company will receive the same level of call quality on your SIP service as an ISDN PRI T1.
https://www.3cx.com/community/threa...but-not-in-the-bye-message.51723/#post-211492

 

[nb]

How is this getting along?
Have we identified the cause of the bad quality audio?

 

[SECOIT GmbH]

Patrick,
From my experience: Don't use G.722 with 3CX. Sometimes it sounds great, many times the quality is awful.
Before 3CX I was using some small router box (Fritz!Box) for VoIP in my office which is popular in Germany and had G.722 activated and it was working well all of the time but with 3CX I never received consistent call quality. I was tracking RTTs, bandwidth, QoS, all perfect, all consistent. But still during roughly 2 out of 3 calls the other end complained.

After switching to G.711a (kind of the defaut in Germany next to 722) all sounds fine and quality is consistent. I changed both the SIP trunks and the phones. I'd love to use "HD" calls (that's what they call G.722 here) with 3CX but that seems it's not quite there yet.
But to be honest... G.711 is in fact pretty much sufficient for calling.

 

[nb]

G722 is the HD codec - so if it sounds awful, it means that it could be a network issue. However the bandwidth it consumes is comparable to g711. So I do not know why you had this negative experience. Just make sure that all devices natively support this codec.

If you want to confirm whether this is 3CX's problem or yours (anything else - fritz or network) then disable PBX delivers audio. This will force the devices to talk to each other directly without 3CX transcoding media. Its a simple trick that completely rules out 3CX. If you still have bad quality, it means you have a problem.

 

[SECOIT GmbH]

Hey Nicky,

Many thanks for looking into it.

I'm using only officially supported devices here in my office and presentation area / lab (Fanvil X4, X6. Yealink T41S, T46S, T48S, T58A, CP860, CP960, W56P. Snom D745, D765, M700/M65, M300/M25. And of course the mobile phone and Windows clients.). I'm sure I did not try all possible combination between these many demo devices but internally they sound great. Also I keep adding new devices to present to my customers so not all of them had been tested with external G.722 calls. But internally G.722 is no issue when the codec is used inside my network (phone to phone RTP with no PBX in between).

Also when I'm connecting some of these devices to the Fritz!Box and use this device for connecting to the outside trunk (Deutsche Telekom) I get perfect results. Bandwith, QoS and latency is no issue, the WAN line is very bored in fact so good results are expected.
It's just when they are connected to the 3CX PBX and I do external calls I keep getting complaints from the other party when using G.722.

Considering with all new provisions with 3CX G.722 sits below G.711 in the codec order list I saw this as the potential reason - I thought by moving G.722 to the top for all devices I wanted to push something which is not quite ready. So I moved all the codecs to their original order and disabled G.722 on the telekom trunks and no longer received complaints.
So really, G.722 is nice but for me it's now show stopper not to use it and G.711a works perfect here that's why I recommended it to Patrick.

But... from reading my own post I guess it makes still sense to at least enable it for internal calls andkeep it disabled on the trunk.
And if you are saying that even the G.722 codecs are not on top of the order it's still "production ready" with 15.5 SP2 I'll might spend some time digging a bit deeper (having a 3CX PBX on one side and a Fritz!Box on the other side and transfer some music, capture the data packages and see if I can narrow it down).
To be fair I believe last time I used G.722 on the Trunk was with 15.0 but considering G.722 is still third rank in the codec list with newly provisioned devices I kind of expected nothing has changed in this area so I didn't bother trying again. If you are saying it's worth it with 15.5 SP2 I'll definitely do.

 

[nb]

I can explain in more detail another interesting reason why you get artifacts in audio quality.
Imagine you have g722 endpoint talking to a gsm endpoint.
Pbx needs to make transcoding. So it has to downsample the g722 stream to 8kHz.
Then it will upsample again on the way out to the codec of the endpoint. This process may cause some audio issues.

I think now we can improve this - we need to downsample the stream for sure but maybe we can use the 16kHz frequency range instead..

Yes try - experiment.. We can discuss your results in this thread... Do the test I told you. If all devices you work with have g722 enabled in SDP, then the offer answer model should be without transcoding.. remove pbx delivers audio so you can experience a direct clean audio negotiation.

 

[SECOIT GmbH]

Hey Nicky,

I'll start doing a list but for now there is one thing I can already say pretty much for sure: When the 3CX PBX is involved in recoding from G.722 to G.711a or in the other direction it sounds... uhmm... Sh... (translated: not so good)

I tried with two different people and different PBX systems (ISDN and SIP). Both reported the same.
I'll continue this list below but what I can already see:
- "Upsampling" the RTP stream from G.711a to G.722 adds a lisp and a "computerish" sound.
- "Downsampling" the RTP stream from G722 to G.711a adds a slight lisp and a strange/unnatural sound
- Using the same config this affects only outgoing calls when the local phone is talking G.722 to the PBX and the PBX recodes it to G.711a. Both on my side and also on the remote side the sound quality is considered low.
- Incoming calls using the tested scenarios are not affected since all parites "speak" G.711a.

I'll keep updating the list when I conduct more tests, also using different PBX, codecs, etc.
Since it doesn't seem to make a difference if the remote side uses ISDN or SIP with G.711a and also if I use a desk phone or the soft client on my end I won't do separate tests for these anymore and continue with only one of them.

 

[SY]

Problematic cases can be easily investigated if wireshark capture of network traffic (taken on PBX host) will be provided.
one of the reson of

- "Upsampling" the RTP stream from G.711a to G.722 adds a lisp and a "computerish" sound​

is the situation where one side of conversation thinks that stream is "PCMA" but other party is sending "PCMU" because think that PCMU is expected on the other end of conversation.
wireshark capture will clearly specify what side of the conversation made mistake.
It is the good subject for support ticket, isn't it?

 

[SECOIT GmbH]

Hi Stepan,
If you look at the columns "Codec Phone <-> PBX" and "Negotiated Codec Trunk" you'll se what codecs actually have been used.

In Germany the "de facto standards" are G.711a and G.722. The values in the column show what wireshark detected.
So... no G.711u involved. To make 100% sure I can remove 711u from all codec lists but I'm already 99% certain it is not involved at all.

I'll remove that codec everywhere and confirm. Still thanks for the hint.

 

[SY]

Sorry, but table does not show how codecs were really negotiated and what was really delivered over network.
The goal is not "to remove codecs to patch problem", the goal is to understand what is going wrong and avoid problem either on 3CX side, or report it to the other side of connection.

 

[SECOIT GmbH]

Sorry, but table does not show how codecs were really negotiated and what was really delivered over network.
The goal is not "to remove codecs to patch problem", the goal is to understand what is going wrong and avoid problem either on 3CX side, or report it to the other side of connection.

Great, got you. Thanks!
I'll do various more tests, add wireshark logs for the connections (both trunk and internal side), finish the table and put in a support ticket so you can investigate.