Random Call Drops Internal & External

[CRTI]

Hi,

Looking for some help/advice from the community.

First issue is the Activity Log is full of the following error

Exception: SipMessage::Exception Missing header Contact @ SipMessage.cxx:1393

Second issue is that calls drop at random. After some digging around last week the system was stable for 3 days but today calls are dropping all over the place. There is no set time that the calls drop, it just happens anywhere between 5 and 30 seconds. Both Internal and External calls are affected.

The 3CX server is cloud hosted, onsite there is an SBC running as a Hyper-V VM, there are 10 Yealink T42S phones connected to the SBC.

Firewall is a Ubiquiti Unifi USG4P with Sip COMMTRACK disabled
Switch is a Ubiquiti Unifi 48port POE switch
Phones all run on their own vlan, latest support firmware has been pushed to the phones

I am rapidly running out of ideas so any suggestions would be appreciated.

 

[Lee Cramman]

Has anything changed recently such as an upgrade? Or has this problem existed since install?

I think these errors relate to a phone trying to register on the PBX with the header information missing.

Might be worth restarting services and your SBC.

 

[CRTI]

The issues have been present since the install 2 weeks ago. I have restarted the System, SBC and phones regularly. Everything seemed stable Thursday/Friday but today the calls are dropping like flies.

 

[YiannisH_3CX]

Hello @CRTI

The first step is to make sure that the firewall checker passes all tests. If there is an issue with the ports that is the first thing to resolve. Then check if the issue affects local calls as well. The activity log of the PBX can provide info as to why the calls are dropping. Have you contacted your provider and check if they can see why the calls are dropping?

 

[CRTI]

Thanks for the info so far guys.

The firewall checker passes no problem. I have not contacted the trunk provider yet as the calls are also dropping internally between extensions.

 

[YiannisH_3CX]

In that case the best way to troubleshoot the issue is through the SBC logs. Assuming the SBC is running on windows you will to enable logging from the config file of the SBC. You will need to navigate to "C:\ProgramData\3CXSBC\" and edit the "3cxsbc.conf" file. Change the "Level" to "VERBOSE" and save the file. You will need to restart the SBC to apply the change.
For linux you need to look into /etc/3cxsbc.conf
Let the issue replicate and check SBC log for any indications are to what is causing the calls to drop. Once you are done troubleshooting revert the changes as verbose logging is CPU and HDD demanding and may cause other issues if left running for too long.

 

[Alphabetic]

Is the call dropping or just audio?

Can you run a wireshark and reproduce the issue?

You can also run wireshark on the yealink handsets themselves.

 

[CRTI]

@YiannisH_3CX I will increase the logging tomorrow and wait for the issue to occur

@kieferschild the call is completely dropping

 

[YiannisH_3CX]

@YiannisH_3CX I will increase the logging tomorrow and wait for the issue to occur

Keep us updated with the results

 

[CRTI]

Update: - Yesterday we had a few dropped calls although less than the day before.

After reviewing the SBC logs at the time 2 of the dropped calls (one morning one afternoon) for both calls there was a gap of 7 seconds in the SBC logs.

The SBC is installed on a Debian 9 VM, the VM is on HyperV on a Dell server with Broadcom NIC's.

Last night I disabled the Time Sync integration of HyperV and installed NTP on the SBC Linux VM. In addition to this I Disabled the VMQ on the VM and also on each of the Brodcom NIC's in the server.

We will see how things go today.

 

[Lee Cramman]

Careful leaving logging on for too long. It chews up system resources and hard disk space.

If you have a spare RPi kicking about you could see if it encounters the same issues as a VM.

Might also be worth checking the specs of the VM. They should really be equivalent to a core i3 with 2Gb of RAM (the required specs for an intel SBC are the same as a PBX, although I suspect a lot of people could get away with less given that an RPi can do it).

Is the VM from a 3CX iso? I ask because the Windows install appears to require a fair bit more resource for Windows to run than the iso does for Debian.

 

[CRTI]

I am clearing the logs each evening so that the space is reclaimed. I suspect the issues are related to the changes I made last night. in which case a Rpi wouldn't experience the issue, but also wouldn't give us the real answer.

the VM is suitably sized and was deployed from the 3cx Debian iso.

So far today we have had no reports of any call drops

 

[Lee Cramman]

Hope your changes have sorted it for good. Maintaining accurate time in a VM is a pain...

With the Pi I was simply thinking that it would tell you if your problem was networking or server based but hopefully it's not necessary now.

 

[YonNomNom]

Update: - Yesterday we had a few dropped calls although less than the day before.

After reviewing the SBC logs at the time 2 of the dropped calls (one morning one afternoon) for both calls there was a gap of 7 seconds in the SBC logs.

The SBC is installed on a Debian 9 VM, the VM is on HyperV on a Dell server with Broadcom NIC's.

Last night I disabled the Time Sync integration of HyperV and installed NTP on the SBC Linux VM. In addition to this I Disabled the VMQ on the VM and also on each of the Brodcom NIC's in the server.

We will see how things go today.

I can confirm that I experienced this on my test setup with the same configuration as you. I also disabled Time Sync and it instantly stopped all my issues with calls dropping and it's been running fine for a few months now.