Debian: 3CXPhoneSystem01 service crashes (randomly) after Update to 15.5 SP5

[Michael Allacher]

Hi,

prior to the update to 15.5 SP5 everything was working stable. However, I just yesterday did the update to SP5 and a few hours later the service crashed with a segfault message.
I also tried the update sometime before in July and had the same issue which was why I reverted my VM to the previous version.
However, the first time I accidentally restarted the system before I got a completion message, so I thought it might be due to that.
This time I did wait until I got the e-mail before doing anything else and I still have the issue, so it seem to be related to SP5.

I got the following error in the syslog:

Aug 18 10:20:04: [55128.256439] iptables[DOS]: IN=ens3 OUT= MAC=XY:XY:XY:XY:XY:XY:XY:XY:XY:XY:XY:XY:XY:XY SRC=192.168.XYZ.XYZ DST=192.168.XYZ.XYZ LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=32260 DF PROTO=UDP SPT=34335 DPT=43022 LEN=1480
Aug 18 10:20:17: [55141.078166] 3CXPhoneSystem[735]: segfault at 3100000006 ip 00007f82fbed3676 sp 00007f82f1dddff8 error 4 in libc-2.24.so[7f82fbe53000+195000]
Aug 18 10:20:17: 3CXPhoneSystem01.service: Main process exited, code=killed, status=11/SEGV

 

[PatrickS]

Hi Michael,

I happen to have a system using Debian 9. The system crashed might be related to memory usage level. A few days ago, we pressure-tested our system for maximum contact (phone book size), using Debian9, and the system process randomly crashed due to memory issues.

After we increased memory, the issue disappeared.

We are running the latest 3CX on Debian 9, and it seems ok.

 

[Michael Allacher]

Hi Patrick,

I don't think it is the memory as the system only uses up to 22% normally and I only updated from SP4 to SP5 and use the same OS (Debian 9) for both. Only since SP5 I have the issue.
However, thanks for your suggestion and I will increase the memory just to ensure that isn't the issue.

 

[Michael Allacher]

After increasing I also ran into the same problem again.
I have now setup auto restart for the service that way everything is wroking again soon after the error again. I hope it will be fixed with SP6 (or maybe after a clean installation which I have to do soon anyway)

 

[Marari]

@Michael Allacher

There's nothing like the smell of a clean install. So fresh. But in this case I think it would be a waste of time.

What you have (it appears) is a segmentation fault.

This is caused by a signal 11 (SIGSEGV) set which is the default action (abnormal termination) when a program receives it.

There are a number of reasons why it can happen such as (but not exclusively):

• Overheating
• Hardware fault
• Memory fault ( bad RAM )
• Faulty driver

Some methods to narrow it down:

• Make sure your memory matches - and most importantly, make sure it matches the system
• Replace sticks of RAM one by one to see if the problem goes away. If the problem returns when you add a stick, you know you have bad RAM
• Turn on core dumping and use gdb to analyze
• Use strace

I hope that guides you in the right direction.

 

[Michael Allacher]

Hi Marari,

thanks for your response. However, I tried downgrading to 3CX 15.5 SP4 and everything worked fine again too. So it is unlikely to be a hardware issue. It might be a driver as in order to downgrade I had to restore a previous state, of course.
Also it ONLY happens with 3CXPhoneSystem01, not any of the other 3CX services.

In any case 3CX 15.5 SP6 just came out and I will try if the issue might be gone after that update.
I also need to move 3CX to a new hardware next week or so soon too, so if the problem still persists it might be gone then. Will keep you posted.

 

[Marari]

@Michael Allacher

The only other thing I can think of is making sure that your Debian 9 install is completely up-to-date.

Best of luck.

 

[Michael Allacher]

Had updated all system components too with the SP5 update.
However, I did the udpate to v15.5 SP6 yesterday and since then the error hasn't re-occured. I hope it stays that way but it is too soon to say with certainty. It also included a kernel update from 4.9.0-7-amd64 to 4.9.0-8-amd64
I hope it is fixed (either due to kernel update or due to SP6 itself) but I let you know if the error re-occured or didn't re-occur after a week has gone by.

 

[Michael Allacher]

Since the update the issue hasn't re-occured, so I guess it most likely is fixed now. There is still the possibility it might happen later on but as it hasn't happened for a week, I would say that is unlikely.