3CX v15.5.15502.6 SIP Server crashing

[mhariri]

I am running on the latest service pack for v15.5 and enterprise license and have experienced two similar 3CXPhoneSystem service crashes in a month. I saw the other posts mentioning the same segmentation fault but they were running on SP2 and the offered solution was upgrading to SP6. Am I now forced to update to v16 because of this although I am reluctant because of the changes it brings and the fact that I do not see anything in the changelogs for recent updates about SIP server bug fixes?

Unfortunately, I did not have verbose logging enabled and I am not intending to do so as this is a production system and and I would not like to cause extra stress and instability to it.

Here are the syslog logs when the crash happened:

Jul 9 10:00:02 3cx kernel: [12838969.484996] 3CXPhoneSystem[5825]: segfault at 0 ip 000000000048f069 sp 00007ff4aa3d6810 error 4 in 3CXPhoneSystem[400000+3d2000] Jul 9 10:00:03 3cx systemd[1]: 3CXPhoneSystem01.service: Main process exited, code=killed, status=11/SEGV Jul 9 10:00:03 3cx systemd[1]: 3CXPhoneSystem01.service: Unit entered failed state. Jul 9 10:00:03 3cx systemd[1]: 3CXPhoneSystem01.service: Failed with result 'signal'.

and more from 3CXConfService:

10:00:03.973|7fe194f2d700| Info|/home/repomaster/workspace/15.5SP6/SPBuild/Sources/Projects/SLDBServ/src/DBServ.cpp(533): SL:Receive failed: socket closed 10:00:03.973|7fe194f2d700| Info|/home/repomaster/workspace/15.5SP6/SPBuild/Sources/Projects/SLDBServ/src/DBServ.cpp(533): SL:Remote disconnect: "3cx:5482/CallManager" stopped 10:00:03.973|7fe194f2d700| Info|/home/repomaster/workspace/15.5SP6/SPBuild/Sources/Projects/SLDBServ/src/DBServ.cpp(72): 3cx:5482/CallManager disconnected

Thanks,

 

[cobaltit]

Care to share anything about your environment? WIndows, Linux, Bare Metal, VM, resources, etc? My first guess is not enough RAM. Could be an early VM (if PBXExpress) that didn't enable swap if Debian

 

[mhariri]

I am running a Debian 9 on an ESXi 6.5 VM on premise with 4GB ram and 4GB swap space. the memory barely reaches 50% usage.

Here is the output for free -h:

total used free shared buff/cache available Mem: 3.9G 1.0G 1.3G 204M 1.6G 2.4G Swap: 4.0G 143M 3.9G

 

[cobaltit]

Oh well then that's likely not the case. There was an issue with the low-end VMs on the cloud with 1GB of memory and no swap crashing, but that was typically just the management console crashing and not the sip service.

I know there was some Debian kernel issues with v6.5 and the recommendation from Vmware was to upgrade to v6.7 or something like that, or there is also a patch. Debian 8 runs fine but Debian 9 crashes. We had another customer with that issue that ended up moving to GCloud so I can't personally verify that upgrading ESXi will fix it but there's definitely something with Debian 9 and ESXi

 

[steev182]

I was having a similar issue with a server on esxi 6.5. Bumped RAM up to 8GB, added a CPU core, it kept happening. I ended up changing the network interface from vmxnet3 to e1000 and haven't had any issues since.

 

[mhariri]

Oh well then that's likely not the case. There was an issue with the low-end VMs on the cloud with 1GB of memory and no swap crashing, but that was typically just the management console crashing and not the sip service.

I know there was some Debian kernel issues with v6.5 and the recommendation from Vmware was to upgrade to v6.7 or something like that, or there is also a patch. Debian 8 runs fine but Debian 9 crashes. We had another customer with that issue that ended up moving to GCloud so I can't personally verify that upgrading ESXi will fix it but there's definitely something with Debian 9 and ESXi

I was having a similar issue with a server on esxi 6.5. Bumped RAM up to 8GB, added a CPU core, it kept happening. I ended up changing the network interface from vmxnet3 to e1000 and haven't had any issues since.

Thank you both for your suggestions! The strange thing is that It was working just fine for a good 7 months in production with the same amount of incoming call traffic and now this has happened.

The only thing that is changed is the number of recorded calls which added up during this time but it is stored on a network share.

I will definitely take a shot at changing the network driver and see what happens if I see another crash like this.