Calls dropping internally and externally

[Keith.vassard]

Hi everyone,

We're having an issue with a customer whose running 16.0.676 as a Linux VM on esxi using mainly Snom 300 phones. All phone firmwares are up to date. The issue started out about 2 months ago when we first installed it as audio dropping during a call for anywhere up to 30 seconds but the call staying active and eventually coming right. This was not every call but a few times a day on both internal and external calls. 3CX support reviewed all our logs and the issue appeared to be time drift. I changed the NTP on all phones and the 3CX to use only one public NTP server. The customer has not said anything for a month but has come back now saying they are again having issues only this time the calls are actually dropping completely. I have begun a packet capture on the PBX and have verbose logging enabled but would appreciate any assistance on what to look for.

Thanks

Keith

 

[JohnS_3CX]

Hi Keith,

This might be unrelated to the previous occurrence. I would firstly start by looking to see who is terminating the call (if it can be seen in the capture)

 

[Spencer Bizzell]

Hi John,
I work with Keith and we are still battling with this issue, he has been onsite at this site this afternoon trying to get to the bottom of this issue but we don't seem to be making progress. It is very odd as we don't have this issue at any other sites.
We have have not had the issues that the customer was referring to where the call drops, we are having the issue which we previously was there.
The calls audio is lost for between 3-5 seconds during a call, and the audio comes back.
We have confirmed it on more than 5 calls today while onsite by playing music on the other side of an extension. The support engineer assisting with our ticket suggesting changing the codecs which we have done to now use the default for the phones which is ulaw, alaw, G722, G729. But we can still replicate the issue.
The PBX is set to deliver the audio on all extensions, and we have confirmed that the time is correct on the PBX. Support re-invite is enabled on all extensions.
This is running as a virtual machine in a Vmware exsi environment which is the only difference to other setups of 3CX we have around as usually we are running them in a Hyper-V environment at all our other sites, but don't see how that could make a difference?
The phones are a mix of Snom phones, we usually run Yealink but the site already owned these phones as it was an upgrade from an existing opensource Asterisk environment, but I wouldn't think this would be an issue with the Snom hand sets.
Any assistance that you could give would be great?

 

[AWS2P]

Who is your SIP trunk provider? what's audio codec set on trunk side? G711U?
Can you try to untick re-invite support on exensions and test call without

 

[Keith.vassard]

Please see attached audio example of what happens at 00:35

Thanks

 

[Spencer Bizzell]

Who is your SIP trunk provider? what's audio codec set on trunk side? G711U?
Can you try to untick re-invite support on exensions and test call without

We are replicating this all on an internal calls. We are using G711u. (We were using G729 before but changed today while testing onsite on the recommendation of 3cx support to G711u)
This doesn't happen on every internal call but happens on most internal calls.
Would re-invite make a call go dead just for a few seconds, we could test this if you think it is needed?
We were informed to turn this on when the problem started, so initially we did have re-invite turned off.
The PBX is set to deliver audio, so the re-invite shouldn't really play its hand should it?

 

[Spencer Bizzell]

I just had a 22min call with Keith who is onsite, and this call did not have the issue so it is definitely not every call. It seems to be intermittent and we can't seem to work out what it is.
We have been running traces and have posted these to Support but they don't seem to be able to pick up the issue at the moment, they are working on it we hoping they will find something soon which will assist.

 

[JohnS_3CX]

Hi Spencer,

Contacting support was the right thing to do in this case. Captures and analysis of logs can reveal quite a bit and Support can advise you where too look or what types of tests to run.

I was wondering so far if your PBX captures contain outgoing audio packets during the time of the silence or if no packets are sent out at that time. I was also wondering what ESXi version you were running at that site. We have some test lab machines on ESXi 6.5 and vmxnet3 adapters here I believe which don't seem to cause any issues in daily operations

 

[Spencer Bizzell]

Hi John,
Thanks yes we have found on one of the calls where the audio drops there is an out of sequence and the bandwidth drops off and there is high latency on the packets at the time.

We are running ESXi 6.7 with ntg3 driver, there was known issues on 6.5 with ntg3 but not on 6.7, would you suggest we change this driver?
We have noticed that there are other guests in the same vswitch which possibly could be causing the issues. We will get the 3CX server onto its own vswitch with a dedicated nic for the PBX only and see if there is any change, should we change the driver as well?
This is a high priority site so we can't afford for these issues to drag on and test too much, as they run the a critical transport system in our city and dropping internal calls has caused an issue in the last emergency.

See below from support:

The weird thing is that they are seeing that we are using G729 codec still but we have changed all the phones and the PBX to the defaults, which are G711u, G711a etc with G729 as the lowest priority. Doesn't really make sense that it is still using the wrong codec, and this has been applied on the PBX?

Any more input or similar experiences would be appreciated.

 

[AWS2P]

The weird thing is that they are seeing that we are using G729 codec still but we have changed all the phones and the PBX to the defaults, which are G711u, G711a etc with G729 as the lowest priority. Doesn't really make sense that it is still using the wrong codec, and this has been applied on the PBX?

after codec change you need to reprovision phones to really use new sorted codec

 

[JohnS_3CX]

Correct me if I'm wrong on this one, but I believe the ntg3 driver controls the physical Broadcom NetXtreme Gigabit Ethernet NIC of your host server so I don't think you can change that driver (unless it has updates available, perhaps update it).

When I asked about the NIC you use, I meant the guest machine's virtual NIC.
Running this command in the VM should tell you what virtual nic you have.
lspci | egrep -i --color 'network|ethernet'


Anyway, it seems like a network issue, so the steps you will take should help alleviate this hopefully if the neighbors are tying up the vswitch too much.

As for the codecs, you need to reprovision the phones otherwise they will use the old codec priority. You can verify this by logging on to the phone UI and check the codec priority.

 

[RemoMeyer]

Hello Keith

Do you have a local or Clioud Instance?

I had this problem with some Clients (Local Instance).

• VMQ in VM Host has a issue and can do this error.
• Time difference zwischen VM Host und VM 3CX can do this error
• No one NTP Server in VM linux can do this error.

This Info did helped me:


https://docs.microsoft.com/en-us/windows-hardware/drivers/network/virtual-machine-queue--vmq-

https://support.microsoft.com/de-ch...lost-on-hyper-v-vms-if-vmq-feature-is-enabled

https://support.microsoft.com/de-ch...work-connectivity-when-you-use-broadcom-netxt