PBX Delivers Audio - poor quality

Discussion in '3CX Phone System - General' started by Frazer, Feb 22, 2018.

Thread Status:
Not open for further replies.
  1. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    Hi,
    We have 50+ Yealink T46S configured with the following codec priority:
    G722
    PCMA
    PCMU
    G729

    Trunk codec priority is:
    PCMA
    PCMU
    G729

    C:\Program Files\3CX Phone System\Bin\3CXPhoneSystem.ini currently set to:
    LocalCodecs = G722 PCMA PCMU G729 GSM
    ExternalCodecs = PCMA PCMU G729 GSM
    3CX Media Service has been restarted since making the above changes, as well as the entire server.

    All phones are local to the PBX, on the same subnet in a dedicated voice VLAN.

    When making internal calls the audio is clearly lower when 'PBX Delivers Audio' is selected.
    S's within speech are particularly poor quality.
    Disabling this on both extensions increases the call quality.

    The Yealink's both show up as a 'HD' call regardless of the above setting, suggesting both peer-to-peer and PBX methods are using G722.

    Any ideas how to achieve peer-to-peer G722 quality while using PBX Delivers Audio or is this not possible?

    Thanks
     
  2. leejor

    leejor Well-Known Member

    Joined:
    Jan 22, 2008
    Messages:
    10,375
    Likes Received:
    231
    What is the reason for using PBX delivers audio? Have you looked through the 3CX activity log to see if there is transcoding involved?
     
  3. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    We may need to record calls.
    The activity log doesn't mention anything like that, although it is set to 'low' - what setting should it be on to reveal transcoding?
     
  4. leejor

    leejor Well-Known Member

    Joined:
    Jan 22, 2008
    Messages:
    10,375
    Likes Received:
    231
  5. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    Currently no extension has 'Record all calls' ticked.

    I've set the Activity Log logging level to verbose and I still cant see any codec details for the call.

    On the phone's display under RTP Status its showing the local and remote codec as G.722, but I imagine the re-encode performed by the server is either reducing the bit-rate with the same codec or doing a poor job at encoding the same bit-rate.
     
  6. leejor

    leejor Well-Known Member

    Joined:
    Jan 22, 2008
    Messages:
    10,375
    Likes Received:
    231
    There won't be any "re-encode" (transcoding) done by 3CX unless a destination cannot support one of the Codecs used in the device initiating the call, or something like recording is enabled. Do you experience the same , poor audio quality if you eliminate the G722 codec rfrom the equation.

    You didn't say why you would want the option PBX delivers audio, enabled when it clearly causes you issues. If disabled, is there an audio issue on external calls?
     
  7. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    I was only surmising that perhaps the 3CX is receiving G722 and for whatever reason, re-encoding it and sending it out as G722 as that is what both phones are seeing, but audio is of lower quality. If 3CX is meant to pass-thru the audio when the codecs match (which seems to be the case) you would expect the same quality audio albeit with a slightly increased delay.

    I mentioned that we might want to record all calls - I was under the impression "PBX Delivers Audio" was a requirement for the PBX to record calls, is this not the case?

    I'll have to double check external calls on Monday.
     
  8. mixig

    mixig Active Member

    Joined:
    Dec 13, 2011
    Messages:
    519
    Likes Received:
    11

    PBX Delivers Audio it does not need to be selected when you recording the calls, only record call check box (and with that, behind the scene 3CX will put it local IP in SIP message for audio just like when PBX Delivers Audio is selected).

    I am guessing that there is no any recoding in your case because your phones both negotiate g722 (you can verify with Wireshark to be sure) in SDP section.

    About poor quality when g722 is negotiate and with proxy RTP stream through the 3CX, maybe 3CX do something locally with that RTP stream which cannot be captured with Wireshark unlike when there is g711 codec (if you tested with g711? and the quality is good?)

    If you select only record call option under the extension settings how is the voice quality then? Probably the same (in this situation recoding is additional step on 3CX I believe).


    Look at the last post from this link from NickD_3CX, maybe will help
    https://www.3cx.com/community/threads/3cx-transcoding-issue.51217/
     
  9. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    I'm unsure how to verify the local/remote/3CX transmissions using Wireshark.

    PBX Delivers Audio is now off on all phones.

    Internal quality - great, verified using RTP status on-screen during the call as G.722 local and remote.

    External quality - acceptable, RTP status shows up as G.722 local and remote.
    Our SIP provider is Gamma.co.uk which only support G.711 and G.729 and I have specified this in the 3CX trunk config as well as 3CXPhoneSystem.ini.
    Call from Three UK (mobile) to 3CX DDI - RTP status shows G.722 local and remote
    Call from Virgin Media UK (land line) to 3CX DDI - RTP status also showed G.722 local and remote.
     
    #9 Frazer, Feb 26, 2018
    Last edited: Feb 27, 2018
  10. lneblett

    lneblett Well-Known Member

    Joined:
    Sep 7, 2010
    Messages:
    2,061
    Likes Received:
    56
    18Am uncertain of the response.

    If Gamma is only supporting g711 and 729, how are you able to determine that any remote side is showing g722? The only way would be if each side was taking the stream and transcoding to 722 from 729 or 711.

    There are very, very few providers that will support g722 as the frequency response that it is capable of exceeds that which most telco and cellular systems can handle. As a result, there is no reason for them to consider it and then burden themselves with the transcoding process.

    Take a wireshark capture of a remote call initiated by one of your phones. My bet is that Gamma and 3CX are using the 711 or 729 codec and that the path between 3CX and the phone is being transcoded by 3CX to 722 to meet the need of the phone as the phone provided the INVITE.

    Repeat the test and then do a call from a remote phone back in to 3CX and see if the transcoding still takes place to 722. In this case 3CX provided the INVITE along with the SDP codec offering and it may be interesting to see what the two negotiate to.

    Keep in mind that the audio can sound no better than the source and that when transcoding up, the process may cause it to sound worse.
     
  11. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    I'm pretty sure our provider does not support G722.
    https://www.gamma.co.uk/products/voice/sip-trunks/ "And as a consequence we are able to support both G.729 or G.711 codecs."

    Our phones have some basic monitoring capability.
    While the call is active you can press More > RTP status which will show on screen details of the call such as delay, jitter, codecs used local and remote etc. This is where it told me the codec information for the calls I tested above.

    I thought the whole point in configuring the Trunk codecs was so that 3CX would negotiate the correct on on behalf of the phone?
    It should prioritise G.711alaw as configured in the trunk settings and the 3CXPhoneSystem.ini but it appears to be using G.722 unless this is a false report.

    I've not used wireshark to capture remote devices before - Do I need to mirror the port of the phone/server with the laptop/pc running wireshark?
     
    #11 Frazer, Feb 27, 2018
    Last edited: Feb 27, 2018
  12. NickD_3CX

    NickD_3CX Support Team
    Staff Member 3CX Support

    Joined:
    Jun 2, 2014
    Messages:
    1,255
    Likes Received:
    63
    Not exactly, the codecs in the SIP Trunk settings in 3CX are only used to negotiate the codec with the provider and in what order. What codecs the phones uses is irrelevant in this decision process.

    The 3CXPhoneSystem.ini file basically only affects the codec priority in the INVITE messages from the PBX to the phone, the reverse scenario is controlled by the order of codecs in the phones interface (what you have set in the Extension Settings).
    The reason why there is a different section for Local and External is have a different behavior for remote extensions (STUN) and different for Local. This does not affect what 3CX will do with Providers.

    I am trying to fully understand the quality issue. I realize it can be hard to describe what you hear in words, but how would you best describe the issue:
    - Choppy audio (gaps in the voice)
    - Degraded audio quality (think, badly tuned radio station)
    - Garbled audio


    You could, but I don't think that is necessary, at least not to begin with. There is a training video we have that demonstrates how to use Wireshark, it doesn't go into very much detail, but it should give you a clue.
    You can skip right to the relevant part here:
    Code:
    https://youtu.be/WutFMNHguGE?t=560
    Note that if you are using Linux, you can make a packet capture right from the Management Console from Dashboard --> Activity Log, then you just download the file and open with Wireshark.
    Mind you if you are using G722, you won't be able to decode as mentioned above, but for a test I would highly suggest switching to G711.

    This surprises me because if A and B are on the same Subnet and they "agree" to use the same codec so no transcoding is required, then 3CX will just forward the packets onto the new IP. Even if recordings are enabled, the audio stream is extracted and transcoded separately, so it still shouldn't affect the actual live conversation.
    What could affect though is the server hardware performance and/or network topology. If e.g. the A and B are on the same switch, but the server is in a completely different part of the network, this could justify your symptom.


    TIP:
    An excellent way of testing the audio quality of individual extensions is using *777. This is an echo extension any of your extensions can dial. Before changing any codecs, call from the extensions, speak into the mic and hear what comes back.
     
    leejor likes this.
  13. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    Thanks Nick for the thorough explanation.
    Our 3CX is running on Windows Server 2016 virtual machine on a Windows Server 2016 Hyper-V host - so I'll leave the Wireshark for now as you recommend.

    The VM is set up with the following:
    Recommended changes made here https://www.3cx.com/docs/installing-microsoft-hyper-v/
    4-core vCPU
    8GB RAM
    127GB HDD
    Virtual interface for DMZ VLAN (external calls/trunk/webclient via firewall/router)
    Virtual interface for voice VLAN (3CX, Yealink T46S all on the same subnet/VLAN, no routers, direct via 10Gb switch uplink)
    Ping times from 3CX to the phones are <1ms at all times.
    I don't see anymore than about 20% usage appear in the 3CX dashboard.

    Regarding the local and remote showing G.722 on our Yealink's RTP status for an external call - I guess this means its G.722 local (phone) and G.722 remote (3CX) to G.711 (Trunk, transcoded by 3CX as per trunk settings). This I can live with as there's no problem with the audio quality in our current configuration.

    When we enabled 'PBX Delivers Audio', although 3CX, the phone and the priority is set to G.722 (confirmed by RTP status during call between 2 extensions) the quality is lower.

    - Choppy audio (gaps in the voice) - no
    - Degraded audio quality (think, badly tuned radio station) - closest match
    - Garbled audio - no

    It sounds like a lower bit rate or sample rate. The S's I mentioned sound a bit distorted.
    I tested this with 'Record All Calls' on for my extension and this triggered the lower quality.
    I've attached 3 second recording in the link below, its supposedly using G722 on both sides, you'll hear the low quality in the S's I've used:
    https://expirebox.com/download/69d16a419acd5774bf35514f8c5f0ce5.html (blue download button)

    I tested with 'PBX Delivers Audio' while dialling *777 - same problem, PBX Delivers Audio on = poor quality, off = perfect.
     
  14. NickD_3CX

    NickD_3CX Support Team
    Staff Member 3CX Support

    Joined:
    Jun 2, 2014
    Messages:
    1,255
    Likes Received:
    63
    In the Hyper-V guide, make sure you have followed the very last section as this can heavily affect call quality.

    In regard to *777, what you are saying is interesting, because regardless if "PBX Delivers Audio" is on, the audio always go to the server (so technically it shouldn't make any difference).

    By the time you have the issue with *777 (echo extension) we can narrow down the problem now.

    Start a packet capture on the Windows Server where 3CX is installed, and make 2 calls to *777, one with "PBX Delivers Audio" off and one with it on, while the phone is in G722 mode. Make sure you hang up the call before stopping the capture.
    Also the recommend test I usually suggest is to count to 10 with a little delay between each number, it is a good audio sample.
    I will send you a PM so you can send me, if you want, the capture so I can check the actually audio quality as it arrives from the network level and how it leaves.
    It is easier for us as we do have software other than Wireshark that can decode G722 from a capture.
     
  15. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    I had a look at the time sync again and it had reverted to time.windows.com - it was set to 'VM IC Time Synchronization Provider' as per documentation when I first installed so not sure why this has changed.
    Tested again with 'VM IC Time Synchronization Provider', same problem.
    I also tried with the following:
    3CX syncing to GPS clock, phones syncing to 3CX - same problem.
    3CX syncing to GPS clock, phones syncing to GPS clock - same problem.

    The documentation seems to contradict itself:
    Code:
    Check if NTP source is set to the Hyper V server
    Open the Administrative Command prompt Click “Start” > Click “Run” > Type “CMD”
    When Administrative Command prompt run “w32tm /query /source”
    The Result should be “VM IC Time Synchronization Provider”
    Then further down it says:
    Code:
    Microsoft recommends to disable build in Hyper-V time sync for for VMs and place the time adjustment into the guest OS it self using NTP for a stable time result.
    Which is it?
    Should time-sync be checked under integration services for the 3CX guest? Should 3CX sync to an External NTP?
    Should the phones sync to the NTP server of the hyper-V host? the 3CX guest? External NTP?
    What is acceptable time drift from 3CX to NTP and from phones to 3CX?

    I'll send over the captures later today.
     
  16. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    4,443
    Likes Received:
    282
    Regarding the documentation, on windows vm's time sync should be set to "VM IC Time Synchronization Provider" as per documentation. The contradicting part of the guide was in fact added after Debian was introduced. Thank you for pointing the contradiction out, we will adjust the guide to resolve this.
     
  17. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    Hi Yiannish
    Thanks for confirming.
    As mentioned by Nick, time sync is important between the phones and 3CX.
    The documentation says that 3CX should be sync'ed to the hyper-v host as above - this is done.

    What is the preferred time source for the phones?
    3CX server?
    The Hyper-V host?
    The Hyper-V hosts time source (domain controllers)?
    Another external time source?
     
    #17 Frazer, Mar 2, 2018
    Last edited: Mar 2, 2018
  18. nb

    nb Support Team
    Staff Member 3CX Support

    Joined:
    Jun 7, 2007
    Messages:
    2,097
    Likes Received:
    142
    HI all - prepared some answers for most points in this thread.. Thank you all for your contributions.

    Phone and NTP
    Phones get their time source from NTP.. Plus the time setting of the phone with respect to 3CX is not important. That is done automatically and provisioned to an NTP server. pool.ntp.org. Do nothing for this - everything is in 3CX.

    timing comment..
    The critical point with timing here is that the HOST VM time drifted from that of the Hypervisor. Thats the timing you need to configure. USe the post above.

    G722 and PBX Delivers audio - I will elaborate in this one.
    However I see in this treat a problem with understanding when it comes to G722 and I want to clarify this and potentially give you some advice on how to use this properly.

    Imagine you have to fit a big box (HD) into a smaller box (PCMU).
    Can you possibly make the smaller box bigger? NO.
    Can you make the bigger box smaller? YES

    And what happens when you do that?
    You are downsampling the big box to the size of the smaller box.
    This means you make it smaller, more inferior and much less powerful than it originally was. Am i right here or no?

    Another example..This is an image of me in my best years :)
    upload_2018-3-6_14-50-23.png
    Can you go from High to Low? Yes you can - You will end up like the left photo.
    Can you go from Low to high? OK you are going to need pixels. From where are you going to get the pixels? Hmmmm I'm going to ask photoshop developers.. :p.

    This is what happens here - And people trying to transcode HD by binding to media server will get this result. So dont be surprised because when this happens in an image, we expect this.

    the reverse... Low to high..
    The media server cannot get the 1's and 0's from the air and compose a better stream.

    You might argue that you can use enhancement software like these mega apps and sites that claim that you dont need to by an expensive camera :) :) :) because their software can enhance a low quality image.. What they do pretty much is they try to guess what might have been in a certain area, and provide you a version of what it “could have looked like” (in the best case scenario), just like slow motion software tries to interpolate missing frames between real frames. What is the result? Ugly artifacts, because it is a guess, not a real captured image. We will not do this in audio..

    You should not be transcoding these streams because you will loose quality.

    You should let it direct. Let the endpoints negotiate between them the best codec they can use between the devices. The sdp offer answer model is there for this reason. Sometimes you get G722. Sometimes you get PCMA.

    If you want G722 across the board, then you should not shoot at 3CX's resources.
    You need to go to your provider and ask him to offer G722. (they will not do this.. not from a network point of view because the bandwidth requirements are the same. But from a transcoding point of view, the effort you need is GINORMOUS.)

    I need you guys to appreciate what happens here. Audio IS just another media.
    The concepts that you guys learnt for images, Are Exactly identical to all media. So Audio inherits the same rules and constraints as any other form of media out there.

    Damages and side - effects
    Now lets look at the damage made or the side effects.. Lets assume that the provider will not use G722.
    As a side effect without transcoding when you count the numbers from 1-10, Everything is clear.

    With Transcoding when you count 1-10 everything is STILL OK except the number 6. "SIX" you hear "SLIX" or a little muffle of the S..It is really not that bad given the "illegal" operation you are doing.. I think a muffled S is okish..


     

    Attached Files:

    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    jed and complex1 like this.
  19. Frazer

    Joined:
    Nov 22, 2017
    Messages:
    25
    Likes Received:
    1
    Hi nb,
    I understand the principals around transcoding, going from high-low and low-high.

    I was not expecting G.722 to be used during external calls. I would expect the external calls to be of lower quality as they are using PCMA with a different bitrate/sample rate/codec so transcoding is necessary from G.722 to PCMA, as per configuration in the trunks.

    The problem is G.722 on internal calls (where bandwidth is not an issue) AND when 'PBX Delivers Audio' is ticked, or when 'Record All Calls' is ticked. The resources available to our 3CX server are quite high and it is up to the customer to decide what resources they can provide to the server.
    Both phones support G.722 as does 3CX - no need to transcode or down-sample the G.722 codec.

    When making an internal call I wouldn't expect any transcoding to be done at all, just pass-thru at the same codec/bit-rate/sample-rate while recording the audio.

    Are you saying _by design_ 3CX will down-sample internal calls when 'PBX Delivers Audio' and 'Record all calls' are ticked regardless of the codec priority?
     
    nb likes this.
  20. leejor

    leejor Well-Known Member

    Joined:
    Jan 22, 2008
    Messages:
    10,375
    Likes Received:
    231
    Reading Vali's comment in this thread...

    https://www.3cx.com/community/threads/codec-priority.53725/

    "I'm not sure, but you could also check if PBX has "Record all calls" enabled - as far as I know, this option also forces audio transcoding like "PBX delivers audio" does."

    Would lead one to believe that using the option does force transcoding. It would seem that clarification, of the function, is required.

    As i asked earlier, unless recording is used on an extension, what is the reason to have PBX delivers audio enabled, when it seem to cause you issues?
     
    nb likes this.
Thread Status:
Not open for further replies.