3CXIVR Server stopped unexpectantly

[jakthree]

3CX v16.0.1.273 Linux on-site

Received this message 8/14/2019 3:06pm
3CX Service notification
Following service(s) were stopped recently:
3CXIVR01

Snippet from syslog.1
Aug 14 15:01:55 wmi-3cx kernel: [5528981.640462] 3CXIVR[17605]: segfault at 7fd524021000 ip 00007fd53fd1020d sp 00007fd523ff89f8 error 6 in libc-2.24.so[7fd53fbe7000+195000]
Aug 14 15:01:55 wmi-3cx systemd[1]: 3CXIVR01.service: Main process exited, code=killed, status=11/SEGV
Aug 14 15:01:55 wmi-3cx systemd[1]: 3CXIVR01.service: Unit entered failed state.
Aug 14 15:01:55 wmi-3cx systemd[1]: 3CXIVR01.service: Failed with result 'signal'.

Snippet from 3CXIVR.log
14/08/2019 15:00:37.263 [000044c6] [PARK] got an active connection (daXqRvYdwF3ysOLnnqcaig..0c0dd5302), waiting for the matching call
14/08/2019 15:00:37.307 [000044c6] [PARK] got a call (-KijQDhNXtC004DLZR5hig..0c8290125), waiting for the matching active connection
14/08/2019 15:00:37.413 [000044c6] [PARK] attached call data to an active connection (-KijQDhNXtC004DLZR5hig..0c8290125)
14/08/2019 15:01:14.736 [000044c6] [PARK] got an active connection ([email protected]), waiting for the matching call
14/08/2019 15:01:54.545 [000044c6] [PARK] got an active connection ([email protected]), waiting for the matching call (last entry before IVR crashed)

15/08/2019 10:37:23.024 [000025c2] STARTING... (I restarted the service at this time)

While IVR was stopped, my users complained that none of their BLF keys would work. For example *641 (force In Office mode) did not work via BLF key or direct entry by keypad. After the service was started the the BLF keys worked.
Let me know if you need snippets from other log files or other info.

What was IVR attempting to do that caused error 6 in libc-20.24.so?
How to prevent crash from occurring again?

Thank you,

 

[AWS2P]

Is your PBX with enough memory ?
Is it on premise ? Cloud hosted?
Debian ? Windows?

 

[jakthree]

3CX v16.0.1.273
Linux ISO (Debian) - dedicated hardware (not VM)
on-site

Memory,

 

[AWS2P]

was it the same 5% use when IVR crashes, do you know usual memory use on this PBX when in highest level of use?

 

[jakthree]

I was not logged into management console (MC) at time of crash. I don't know how to log memory usage. I have never seen memory use go above 30% but that is right after I login to MC. Typically the memory use is less than 10% even during high use. I only have 30 extension, 2 bridge and 2 trunk (FXO gateway). Typically no more than 4 simultaneous calls (SC). Licensed for 8-SC. Most of the time only 1 SC. It is not a heavily used PBX.

 

[AWS2P]

Ok, so your hardware ressources seems to be oversized to do the job , so this should exclude missing memory scenario, except if memory modules have failure , did you check modules doing a memory test ?

why having 30Go of memory if this is a small PBX? perhaps 8Go is enough.

 

[YiannisH_3CX]

Can't really say why the service crashed from the data provided. To determine exactly what caused it you will need to probably have the full suite of 3CX logs in verbose and perhaps the event viewer from windows to see if there was an error during that time.
If this was the first time this happened you should keep an eye on it as it may have been an one time occurrence. If this happens again however i would recommend creating a ticket with our support department so they can look into the issue in detail.
The first course of action is to upgrade your PBX to SP2.

 

[jakthree]

This was a one time event. The server did send me an e-mail when it occurred, but I wasn't concerned at the time because I have not implemented IVR. For the future, I now know what problems are caused when the service stops and know how to restart if it happens again. I will upgrade ASAP, keep an eye on it and advise if service crashes again.
Thank you for the help.