Audio dropout and several register/unregister event on many extensions

[SimonLessard]

Hello,

I just deployed 3CX v16.0.0.1581 after some test with a few stations but I am having critical audio dropout and some weird register/unregister events.
I used the distro built with Debian on a VM. 30 Grandstream GXP2135 phones, all local. We have a main switch connecting all our network together as well as a lot of small switches in each of the separate (local) rooms.
I was using OPUS codec at first but reverted to G.711U. No major change.
Audio dropout happen on internal and external calls.

1- Should I worry about the many register/unregister events? For many extension, it happens every 3-5 minutes.
2- I'm playing with Wireshark (very novice) and I see some packet loss on the RTP stream. What else can I look for to further investigate?
3- Should I mess with QOS?
4- Any suggestion on how to further troubleshoot this and improve the call quality? Sometime it is so bad, we can't hold a conversation (even in local, over the next room).

Thanks,
Simon

 

[cobaltit]

VM specs? VMware or HyperV? Did you follow the guide?

 

[Lee Cramman]

One possibility:

VM networking problems pop up on here from time to time, mainly to do with how the VM emulates process offloading for the vNIC. I can't remember 100% but I think this was VMWare specific and didn't affect HyperV.

A symptom of this is very high processor use on the host (but not the VM).

If it is this then the easiest fix is to try a different vNIC in the VM's confg.

 

[JohnS_3CX]

Hi Simon,

Just out of curiosity, if you use the webclient do you still have the same issues?

 

[SimonLessard]

Thanks everyone for your quick reply. This really helps since there is a kind of panic here

Cobaltit : we use VirtualBox, latest version on an Ubuntu 18.04 LTS host. I don't think we followed any specific guide, it was pretty straightforward to create a VM and drop in the .iso

Lee Cramman : we do not observe a high CPU activity on the host. When you say "try a different vNIC in the VM's confg", a vNIC is a virtual NIC, right? We have two physical NIC on the host, one for local network, the other for the VOIP channels, coming from a modem from the VOIP supplier.

JohnS_3CX : someone used the webphone on Android and it was worse on the Wifi than on her LTE mobile network. Well, in this case the wifi may be the problem too... I will do some tests with the softphone on a wired windows machine.

On the register/unregister of extension topic, something still bothers me : in the unregister event, the port specified is 5060. Which is obviously the default SIP port. However, I changed the default ports upon installation (we had so many foreign connection attempts with the default ports). So I am worried when I see that something is trying to do something on the port 5060. Am I right?

SIP Server/Call Manager ID: 4101 Extension 139 is registered, contact: sip:[email protected]:5483;rinstance=714c5dd93a0e1363, received from: 127.0.0.1

SIP Server/Call Manager ID: 4101 Extension 139 is unregistered, removed contact: sip:[email protected]:5060;rinstance=1-0de825b1-3deb-4c8f-960c-69347dfd3fc4;inst="513bab9a", received from: 127.0.0.1

 

[JohnS_3CX]

specified is 5060. Which is obviously the default SIP port. However, I changed the default ports upon installation (we had so many foreign connection attempts with the default ports). So I am worried when I see that something is trying to do something on the port 5060. Am I right?

Your PBX might be listening to a different port, but the agents will be listening to their own ports, that's normal to not have all the devices listening to the same port. (i.e my PBX listens to 5060 but my deskphone to 5062)

 

[Lee Cramman]

Did you choose a Debian 64 Bit VM in Virtualbox?

Have you checked that your primary codec is G.711U all the way through (i.e. the extension, user and settings / general)?

Do you have any other VM's running on that host?

 

[JohnS_3CX]

You can also disable PBX delivers audio so that local extensions can reach each other directly for internal calls (under Extension > Options).

Do you still have the issue in this case?

 

[SimonLessard]

Yes, 64 bits Debian in Virtualbox.
There are other VM running on that machine but we are still early in the deployment so there is not much traffic yet. Plus, we are a 30 employees company, I doubt that we can overload that brand new server.

Yes, I downgraded everything to G711U. With Wireshark, I can confirm that by looking at the SIP negotiation packets. Originally, I aimed OPUS but I went back to G.711U.

 

[SimonLessard]

PBX deliver audio is already disabled on the extensions. I can confirm in Wireshark, looking at the RTP stream, packets won't transit by the PBX.

 

[Lee Cramman]

There are other VM running on that machine but we are still early in the deployment so there is not much traffic yet.

I was thinking more about verifying that the network config works OK for other VM's.

 

[SimonLessard]

Yes, 64 bits Debian in Virtualbox.
There are other VM running on that machine but we are still early in the deployment so there is not much traffic yet. Plus, we are a 30 employees company, I doubt that we can overload that brand new server.

Yes, I downgraded everything to G711U. With Wireshark, I can confirm that by looking at the SIP negotiation packets. Originally, I aimed OPUS but I went back to G.711U.

 

[SimonLessard]

I was thinking more about verifying that the network config works OK for other VM's.

Ok I see. What kind of bad configuration could cause the VMs to interfere?

I'm beginning to think that maybe my network is in bad health. I don't know how to verify that but what if a network equipment was causing significant packet drops?

 

[cobaltit]

VirtualBox is not a supported hypervisor.

 

[SimonLessard]

VirtualBox is not a supported hypervisor.

Since the calls are endpoints to endpoints, does it really matter?

 

[SimonLessard]

Also, there is a guide on 3CX on how to set it up with VirtualBox : https://www.3cx.com/docs/debian-linux-using-virtualbox/
They provide instructions so I guess it is supported?

 

[JohnS_3CX]

In the introduction be we note this should be used for testing purposes. Do not install 3CX with virtualbox for productive environments.

 

[SimonLessard]

Wohha, well noted! Thanks for this, I will correct that.
But sorry to insist, since the calls are endpoints to endpoints, does it really matter?
Can it be the root cause of my audio dropout?

 

[JohnS_3CX]

Well, removing PBX delivers audio makes it endpoint to endpoint so I don't think it should matter (once confirmed via capture too).

Either the network is messing with the phones or the phones themselves are facing some issue. You detected packet loss however, so I would not suspect the phones first. Perhaps you can move them on a separate network switch if you have power adapters, with fresh LAN cables and to see if it will persist. Try to isolate the components one by one until only the phones remain.

A capture from the phone interface should also give some more insight as to what the phone sends and receives.