Voice Latency with with PBXDeliverAudio

[Kris2k]

Hi everyone,

I'm preparing to converge my company (20 sets / 8-10 channels) on 3CX, and I'm running a 3CX (the recent release) in a dedicated VOIP vSphere 4.0 U1 Hypervisor to which I have literally dedicated 2 cores to my Server 2008 R2 installation with physical core affinity (yes, I was a little paranoid on ressource allocation).

Now, whenever I stream RTP through 3CX from my gateway that's 10ms away, I feel as if 3CX is taxing almost 100-150ms of voice-latency to the RTP stream, making user-experience really problematic. I'm currently still trying to assess the hard-facts on this, however, there is a perceivable latency that makes the call 'uncomfortable' for an end-user.

The issue is not really an issue, since I can re-invite my peers and steer 3CX away from the Media-Path, however, I will probably want to do some ULAW-ILBC transcoding for my Polycom Phones, and even some RTP proxying for certain peers because of their network particularities that won't support re-invite.

Are there any known parameters for the 3CX mediaServer with regards to buffering the RTP stream? Or anything that could be adjusted on at this level?

Cheers
Kris

 

[nb]

Hi

do not troubleshoot this in the way you are doing. Leave everything default, codecs and all.

TIP TEST 1: Change to 1 core and see if the problem resolves before applying this article.

If yes, change back to 2 cores and read / apply this article
http://wiki.3cx.com/documentation/troubleshooting/stuttering-voice-quality-in-hyperv-and-vmware

This is because of PMTIMERS on virtualized machines and 2 cores
Let me know how this goes.

 

[Kris2k]

Good approach; I'll scale-back to one core once I solve my current multi-homed RTP issue with 3CX.

However, given that I'm using 2008 R2; the use of /usepmtimer doesn't seem applicable.

http://blogs.technet.com/perfguru/archive/2008/02/18/explanation-for-the-usepmtimer-switch-in-the-boot-ini.aspx

 

[nb]

Yes this is not applicable to 2008 OS. i knew this - I probably skimmed part your message and did not take note of the os.
If you have 2008, for sure this is not applicable.

Then 1 question - does the hyper V have enough resources?

Can you explain this

physical core affinity (yes, I was a little paranoid on ressource allocation).

What did you do here?

Because this is the only thing i can think of at the moment.

 

[Kris2k]

I'm using ESX/vSphere4U1, which allows you to statically choose the cores you wish to operate, and literally subscribe the desired ressources; it doesn't make any apparent difference with or without, so I've dropped the parameter.

I think I found the issue while scanning through some 3CX documentation, since I kept noticing that 3CX kept establishing the two RTP peers in Transcoding mode, which means that it takes the G711, transcodes it into a native (internal) format, and then recodes it into G711. Meaning that it has to undertake it's own jitter-buffer for both endpoints, and that alone could probably count for about 30-40ms / endpoint of voice latency.

Apparently, 3CX also does RTP Proxying mode when PBXDeliverAudio is set to No; which means it acts as as imple RTP router if it's unable to find an RTP bypass solution with both peers supporting ReInvite. So that's what I did; turned-off PBX Deliver Audio on my gateway and made sure it didn't support Invite/Options (since its really the case), and tried a call.

It still bridged the call in Transcoding Mode; I made sure that G711 was the only supposed codec for the Gateway, and I updated my Polycom to only support G711 with its codec prefs having the other codecs set to None (via manual provisioning), and tried again. Still no luck; still doing transcoding for inbound and output calls.

Is this a bug? I'm waiting for my reseller to deliver my 3CX so I can post this in the real support forums, but for the time being, I'm ironing-out the technicalities of the platform with the 2 channel demo/feature license.

 

[SY]

Hi Kris,

Media server modes are described in article http://wiki.3cx.com/documentation/architecture/media-server
Some of devices have overridden set of capabilities. It is done basing on the results provided by careful testing of the devices (all call scenarios should work as expected). So, options, which are configured in management console ("PBX delivers audio", "Supports re-invites", "Support replaces"), may not be applied (or partially applied) to specific device.

PBX prints "Device Info: ..." message in the log for each call participant (device joined to the call) and specifies default set of options overridden for the device.

In addition to information provided above:
Extension with SRTP enabled always work in transcoding mode
Record all calls for extension - the same. Always transcoding mode.

The "transcoding mode" is applied in case if it is really required or requested by user.

Regards

 

[Kris2k]

Hello Stepan,

So how can I be sure that 3CX is behaving how I would want it behave?

Evidently, the device-info is properly recognizing my Polycom Phone, but its not recognizing the UA coming from the Class-5 Telco switch (Versatel), which is now leaving me to assume that whenever 3CX doesn't recognize a UA, it forces the most 'conservative' approach to handling the RTP stream?

Target endpoint (deliver-audio=n, reinvite & options = y) residing on ip zzz.zzz.zzz.zzz:
16:39:24.536 [CM505001]: Ext.343: Device info: Device Identified: [Man: Polycom;Mod: SoundPoint IP Series;Rev: General] Capabilities:[reinvite, replaces, unable-no-sdp, no-recvonly] UserAgent: [PolycomSoundPointIP-SPIP_650-UA/3.2.2.0477] PBX contact: [sip:[email protected]:5060]

Source endpoint (deliver-audio=n, reinvite & options = n):
16:39:23.928 [CM505003]: Provider:[Unlimitel-GatewayA] Device info: Device Not Identified: User Agent not matched; Capabilities:[reinvite, replaces, able-no-sdp, recvonly] UserAgent: [Tangerine-oSIP-UA/1.06] PBX contact: [sip:[email protected]:5060]

Hmm, now which XML/INI/Database-entry should I go take a peek and override its 'conservative' behaviour at my own risk?
Kris

 

[SY]

Hello Stepan,

So how can I be sure that 3CX is behaving how I would want it behave?

Evidently, the device-info is properly recognizing my Polycom Phone, but its not recognizing the UA coming from the Class-5 Telco switch (Versatel), which is now leaving me to assume that whenever 3CX doesn't recognize a UA, it forces the most 'conservative' approach to handling the RTP stream?

Target endpoint (deliver-audio=n, reinvite & options = y) residing on ip zzz.zzz.zzz.zzz:
16:39:24.536 [CM505001]: Ext.343: Device info: Device Identified: [Man: Polycom;Mod: SoundPoint IP Series;Rev: General] Capabilities:[reinvite, replaces, unable-no-sdp, no-recvonly] UserAgent: [PolycomSoundPointIP-SPIP_650-UA/3.2.2.0477] PBX contact: [sip:[email protected]:5060]

Source endpoint (deliver-audio=n, reinvite & options = n):
16:39:23.928 [CM505003]: Provider:[Unlimitel-GatewayA] Device info: Device Not Identified: User Agent not matched; Capabilities:[reinvite, replaces, able-no-sdp, recvonly] UserAgent: [Tangerine-oSIP-UA/1.06] PBX contact: [sip:[email protected]:5060]

Hmm, now which XML/INI/Database-entry should I go take a peek and override its 'conservative' behaviour at my own risk?
Kris

Kris,

there is one problem. You didn't specify your expectations...

Thanks
P.S. I mean, do you need to everything work in "by-pass" mode and accidental problems with call signaling (absence of music on hold, unsucsessful transfers, call disconnecting) will not be the problem on your environment?

 

[Kris2k]

Hello Stepan,

I've read (and re-read several times) the MediaServer behaviour relating to RTP Transcoding versus Proxying, and I think I came-up with a clever approach of altering the LocalSubnet configuration parameter to define non RFC1918 subnets, and basically altered what 3CX thinks of my network topology, as well as activating Re-Invite on my gateway in order to leverage the functional requirements of RTP-Proxy even though we really don't want to re-invite.

My initial LocalSubnet declaration was 0.0.0.0/0, because in my case, NAT is non-applicable

Anyhow, it works! My 3CX Internal Interface subnet, as well as my carrier gateways are now all part of the LocalSubnet declaration, and my SIP Phone endpoints are all considered 'Public' even though they transit through private links using Public-IP addressing.

Therefore, the topology requirements for RTP-Proxy eligibility are respected:
- A call flow traversing internal & public networks
- Both peers support re-invite, but because we are going from Internal->External, the RTP proxy is required to leverage the NAT issue which is not applicable in my case.

I think the core-issue is that 3CX tries to be 'Too NAT friendly' and it presents a slight usability issue for configuration, but also, I think the admission rules for RTP-Proxying are a little superfluous. If both peers are willing to speak the same codec, why engage transcoding?

Anyhow; RTP-Proxy solved my latency issue , and comparative testing has demonstrated that my call transiting through 3CX, and my call transiting through my legacy Asterisk system, both exhibit the same call latency now. When transcoding was active in 3CX, there was easily a 200ms tax, which is most-likely because of the Hypervisor not being able to provide an accurate high-resolution clock (which is the common problem for realtime protocols), or else, a too high amount of jitter-buffer within 3CX for the transcoder?

 

[SY]

Kris,

I can repeat: Transcoding mode is applied in cases if it is required for reliable call flow. It doesn't depend on network topology or specific codecs used by endpoints. Is it specified in article?

About lattency added by transcoder.
Ideal transcoder should provide:
1. 0ms lattency
2. 0% of CPU usage
3. quality (not only jitter) of incoming stream compensated 100%

3cx provides only an approximation of ideal transcoder. As far as transcoder has some drawbacks, PBX tries to provide you ability to avoid it but not in all cases.

 

[Kris2k]

Stepan,

I understand that it's to ensure call-flow reliability, however, if my SIP Gateway/Phone doesn't support RE-Invite (either because of functional restrictions/Network Topology restrictions), I should be able to disable the ReInvite option, and expect 3CX do to RTP proxying, instead of getting involved with costly Transcoding if both call-legs are willing to speak the same codec!

This is the key issue; transcoding costs a significant usability issue when the operating system is virtualized; the core deficiency of 3CX-Transcoding is a common issue for all other realtime-protocol platforms out there; the RTC is just isn't precise enough, and directly-affects the performance of the transcoding.

Now, after experimenting further-more with only declaring my TelcoSwitch within the LocalSubnet, it solved the key latency-issue, however created another. Home-Office to Head-Office calls can't get done, because 3CX want's to go into RTP-Bypass mode, and essentially re-invite the call, which is problematic due to network topology constraints, and because I seem to be stuck to leave ReInvite=Y and leverage LocalSubnet declarations, because as soon as I disable this option, it forces transcoding on me.

In a simplistic world; here's how I would have configured everything:
- My PSTN gateways having ReInvite=N, Options=N, PBXDeliverAudio=Y, because my PSTN gateway doesn't like re-invites.
- My Home-Office Phones having ReInvite=N, Options=N, PBXDeliverAudio=Y, because I don't want to worry about network Topology issues.
- My Head-Office Phones having ReInvite=Y, Options=Y, PBXDeliverAudio=N, because I don't want to double-haul RTP via 3CX for nothing
LocalSubnet declared covering PSTN Gateways (non-RFC1918), and Head-Office Subnets (RFC1918), because they are indeed Local without any NAT firewalls .

If a HomeOffice Phone calls the Head-Office Phone (or vice-versa), and there is a common Codec between two call-legs, RTP-Proxying is in effect, or else RTP Transcoding is done.
If a HomeOffice/HeadOffice Phone calls the PSTN, and there is a common Codec between two call-legs, RTP Proxying is in effect, or else RTP Transcoding is done.

Your input is greatly appreciated!

Thanks
Kris

 

[N_G]

Its imperative you use supported hardware. In your post you do not specify which gateway you are using, just that it does not do re-invites. The first thing you must do is replace that hardware with supported gateways - we support the market leading models.

 

[Kris2k]

Its imperative you use supported hardware. In your post you do not specify which gateway you are using, just that it does not do re-invites. The first thing you must do is replace that hardware with supported gateways - we support the market leading models.

But the issue here is not about supported gateways, its about disabling the ReInvite option (a feature which is made available by the product), and noticing that 3CX always systematically goes into Transcoding mode, when it should be doing Proxy Mode because both call legs are willing to speak the same codec. And because of 3CX, your product is taxing between 100-120ms of voice latency, when if it worked properly, latency would have been less than 10m due to the nature of RTP proxying not requiring any packet assembly-disassembly. Its purely a design issue which was probably done on the basis of reliability (which is a good move), however, never fully taking in account the virtualization factors that are at play.

I wouldn't have any problems changing the Call switch, if it didn't cost nearly 200k, and handles a few several thousand channels.....

 

[N_G]

Actually our PBX works fine, the environment and equipment you are proposing is unsupported. Unfortunately I believe we have to agree to disagree!

 

[mit]

I have a couple of observations on this having raised a very similar topic recently.

Firstly, it seems hard to make people understand that a 100ms latency on a call is a problem. The point is that while this latency is tolerable, it significantly reduces the user experience, particularly when you are trying to convert users who are coming from a legacy TDM PABX, and where the predominance of traffic is to local destinations.

Secondly, in my particular case, the only solution was to resort to using re-invites to bypass the media away form the PABX, however, this did not work as transfers no longer worked correctly. This appears to be due to the fact that the 3CX is not correctly complying with the SIP rfc's in relation to reinvites. In this instance, the response of "use a supported VoIP provider or hardware manufacturer" is not helpful. It is not always practical for people to do this, and it would be far better if the 3CX server properly respected the RFC's.