3CX intermediately stops to respond to SIP messages (restart required)

[Olle]

We have a 3CX witch, after an upgrade from 15.5 to 16.0.3 intermediately stops to respond to SIP messages. The system is large, so we do not have much time to trouble shoot when it happens.

The problem is resolve if we restart the server. We have tried to restart services and that did not seem to solve it, but we are not completely sure about that jet. The system has frozen 6 times so far. The first two times it was a couple of weeks between the problems. And now a few days or a week.

We are in contact with 3CX and they have looked at verbose system logs and sip logs without finding anything. However, now 3CX say that we have to factory reset all phones and use a default template, which would be fine except for the fact that we then manually must reconfigure +1000 extensions.

Below is a small snippet from Wireshark. The strange part is that the SIP server (10.102.16.10) responds sometimes. All REGISTRATION and INVITE is answered with 100 Trying and then times out. Still the server is responding to CLASSIC-STUN (keep alive request from a Grandstream phone).



Have you experienced anything similar?

Regards,
Olle

 

[JohnS_3CX]

Hi Olle,

For now I would suggest to follow the instructions give by support and follow up with them after testing to see how it goes.

I can make a suggestion for now. You can potentially disconnect the phones centrally via your switch, and connect only a small batch to test. If the issue appears with the small batch (replication succeeds) then reset/reprovision that small batch after restarting the server and see if the issue goes away. Then decide how to move forward accordingly.

 

[Olle]

Hi John,

We will definitely follow the instructions we got from 3CX, however to factory reset, re-provision and manually configure the settings of 2000 phones is expected to take at least 100 hours. And we have to make sure that we don’t cause any other issues. Anyway, we are preparing to start with this asap.

The problem have only happen during business hours. Disconnecting most of the phones during business hours will not be accepted by the customer (besides that sometimes the system works for several weeks).

We are desperately trying to get a hint of the root cause and so far we do not have a clue. Everything’s seems to be working and no errors anywhere. The only problem is that the SIP server stops to respond on most SIP messages.

Thanks for your reply. We appreciate all help we can get.

Regards,

Olle

 

[Olle]

Here is a more clear picture of the sip traffic when the problem occurs. The packedge dump is from the server 3CX is running on. The only response we get from 3CX after the error is 100 Trying.

Restarting all 3CX services does not resolve this problem
Yesterday we tried to resolve the problem by restarting all services from 3CX Web console. All services except 3CX PhoneSystem 01 Queue Manager Server (that service was not restarted, pending restart) was restarted. The we tried to restart 01 Queue Manager Server directly from the Windows server, but that did not work either.

Restarting the server resolves the problem
If we restart the server it goes back to normal operation.

 

[JohnS_3CX]

You can do 3CXStopServices && 3CXStartServices from command line on Debian.

 

[Olle]

You can do 3CXStopServices && 3CXStartServices from command line on Debian.

We use Windows Server 2016. But if we get completely stuck we consider to migrate the system to Linux.

 

[JohnS_3CX]

Sorry Olle, in this case then stop the services and start them from the windows service manager or task manager services tab.