Several Debian PBX's stopped working overnight

[NZvoip]

First post, so please bare with me.

We currently have ~20 clients using 3CX systems on a variety of operating systems. We have 6 that run on Debian and were setup with the 3CX provided image and the rest on Windows 7, 10, or Server 2012 R2. We've only had the odd issue with 3CX itself, most of the problems come from the network or the environment. We also have Pulseway installed on all PBX's for monitoring and basic remote management.

This morning I was contacted by a couple clients advising their phones weren't allowing calls out - Grandstream phones said 'Not registered', Fanvil phones said 'SIP register is down'. The affected systems I could browse to the 3CX management page but couldn't login. 'Server error' was the only message i received. I rebooted the first and second affected PBX's (via pulseway monitoring agent) and within a minute everything was working again. I then checked each client's PBX by logging into the management console remotely. Turns 3 more of the Debian PBX's (5 out of 6) had the same issue and all the Windows based PBX's were fine. I couldn't really do much troubleshooting as I needed the systems up and running and with my limited linux knowledge I could only run a few basic commands via our pulseway monitoring.

They have been working well for the last year or so, without issue, so much so that we're looking at moving the rest of our clients to linux.

Here's more information regarding all the affected clients:
Using ESX VMs or physical machine
Uptime varied from a few days to months
Management console login page loaded on all of them
Same error when trying to login, 'server error'
Behind a variety of firewalls
Using a variety of ISPs
Using different trunk providers - ufone / vocus
Outgoing calls failed
Internal calls seemed to fail (going off customer information)
Incoming call failed
All located in NZ
All on NZ timezone
Event logs stopped at various times for the different client - 9:30pm / 11:30pm / 12:00am
Event logs stop on random logs - extension register, logs/temp files deleted, etc
Automatic updates are turned off
Only outstanding update is the v16 alpha

I'm looking at improving our monitoring so it will alert if a single service stops on linux (already does this for windows).

Has anyone else experienced a similar issue or can shed some light?

 

[YiannisH_3CX]

Hello @NZvoip

Please note that we have not received similar reports by other users so not sure why this happened to multiple systems at the same time. I would recommend creating a ticket regarding this with our support department so they can look through the PBX logs and try to determine the cause of the issue.

 

[complex1]

Hi,

I know there is an Debian update to v9.8
After the update the system must be reboot, if not you get problems with the system, such as "Server error"

 

[NZvoip]

Thanks YiannisH_3CX,
I have logged a support ticket and have just updated it with some more information that I've found.

Thanks complex1,
We had one Debian instance that didn't have the issue and I found that postgres was updated to v9.6.11 on the affected PBX's but was still at v9.6.10 on the unaffected one.
Ironically, the unaffected PBX was the only one with 'automatic updates' enabled within 3CX.

I'm wondering if there is a system cron job that automatically updates Debian or some components regardless of any settings in 3CX. Just waiting to hear back from 3CX support.

 

[YiannisH_3CX]

Since you have a ticket we should be able to see more details regarding the cause of the issue. The PBX will only install critical security updates if the option is enabled under the PBX update settings.