3CX Debian VM Crash

[DaveB-Opt]

Our Debian VM with 3CX occasionally crashes.
I've pasted the last entries from syslog below.
The clients stop responding, as does the web portal (portal shows server error)
It's currently running on Hyper-V Server 2012 R2
4 processors, 4GB RAM
We never have any more than 4 simultaneous calls running.
Any ideas where we can start to diagnose?


Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] Call Trace:
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb3e1a979>] ? __schedule+0x239/0x6f0
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb3e1ae62>] ? schedule+0x32/0x80
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb3e1e187>] ? schedule_timeout+0x167/0x380
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb38ea640>] ? del_timer_sync+0x50/0x50
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb38e3c35>] ? rcu_gp_kthread+0x505/0x850
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb38bea4f>] ? __wake_up_common+0x4f/0x90
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb38e3730>] ? get_state_synchronize_rcu+0x10/0x10
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb389be69>] ? kthread+0xd9/0xf0
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb3e1f871>] ? __switch_to_asm+0x41/0x70
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb389bd90>] ? kthread_park+0x60/0x60
Feb 17 11:48:33 OMG-3CX-VOIP kernel: [390500.476076] [<ffffffffb3e1f8f7>] ? ret_from_fork+0x57/0x70
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: systemd-logind.service: Processes still around after final SIGKILL. Entering failed mode.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: Failed to start Login Service.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: systemd-logind.service: Unit entered failed state.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: Stopped Login Service.
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: Starting Login Service...
Feb 17 11:48:41 OMG-3CX-VOIP systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE

 

[Lee Cramman]

On the assumption that the VM is running Debian (because why wouldn't you):

I would start by validating your install / virtual hardware settings against this https://www.3cx.com/docs/installing-microsoft-hyper-v/

Pay particular attention to the sections on static MAC, physical NIC driver, virtual NIC choice, using the correct generation of VM and installing NTP.

If that doesn't work you can spare the short downtime I would be tempted to take a snapshot, reinstall to the same VM / VHD from the most up-to-date 3CX ISO and then restore from backup. If everything goes well you can (and should!) remove / integrate the snapshot afterwards to save yourself some potential pain in the future.

 

[DaveB-Opt]

On the assumption that the VM is running Debian (because why wouldn't you):

I would start by validating your install / virtual hardware settings against this https://www.3cx.com/docs/installing-microsoft-hyper-v/

Pay particular attention to the sections on static MAC, physical NIC driver, virtual NIC choice, using the correct generation of VM and installing NTP.

If that doesn't work you can spare the short downtime I would be tempted to take a snapshot, reinstall to the same VM / VHD from the most up-to-date 3CX ISO and then restore from backup. If everything goes well you can (and should!) remove / integrate the snapshot afterwards to save yourself some potential pain in the future.

Thanks - I followed the Hyper-V installation tutorial when I first set this up so I haven't missed anything there.
I'm happy to do a rebuild - just wondered if I can diagnose the current issue

 

[JohnS_3CX]

Hi,

Since systemd-logind.service can't even start, it would seem that the OS or host is corrupted, something that is unrelated to 3CX at that point in time.

I would take a 3CX backup as soon as possible and redeploy on a fresh VM.

You should also consider inspecting the machine itself if the fresh VM also keeps crashing on you.
Use the latest 3CX ISO, not one you have from before (download links in your customer portal)
Make sure you do not skip anything from the guide, especially the NTP part:
https://www.3cx.com/docs/installing-microsoft-hyper-v/

 

[DaveB-Opt]

Hi,

Since systemd-logind.service can't even start, it would seem that the OS or host is corrupted, something that is unrelated to 3CX at that point in time.

I would take a 3CX backup as soon as possible and redeploy on a fresh VM.

You should also consider inspecting the machine itself if the fresh VM also keeps crashing on you.
Use the latest 3CX ISO, not one you have from before (download links in your customer portal)
Make sure you do not skip anything from the guide, especially the NTP part:
https://www.3cx.com/docs/installing-microsoft-hyper-v/

Thanks - we backup daily so I'll spin up a new instance.
Everything was followed including NTP

 

[JohnS_3CX]

Yep and see how it goes from there. If the server is dedicated to 3CX you could deploy the iso there directly and skip hyper-V, otherwise consider wiping the host and getting rid of 2012R2 since its EOL now too and installing 2016 or newer.

 

[DaveB-Opt]

Yep and see how it goes from there. If the server is dedicated to 3CX you could deploy the iso there directly and skip hyper-V, otherwise consider wiping the host and getting rid of 2012R2 since its EOL now too and installing 2016 or newer.

Thanks - we do plan on upgrading the HV soon.
It's re-installed and restored so let's see how we get on.
Can I update this box using SSH just like a regular Debian instance or should I only deploy updates from the web interface?

 

[JohnS_3CX]

Hi Dave,

Once you deploy our ISO, the rest should only be done via the managment console.

There may be some dependencies that should not be manually upgraded as they could inadvertently damage the PBX system, or cause corruption that may go unnoticed until it is too late.

 

[Lee Cramman]

Thanks - we do plan on upgrading the HV soon.
It's re-installed and restored so let's see how we get on.
Can I update this box using SSH just like a regular Debian instance or should I only deploy updates from the web interface?

Security updates are handled by 3CX. PBX updates are handled via the web console. You shouldn't be attempting any command-line updates ( a huge benefit of using the ISO IMO).

 

[ChrisC_3CX]

just to clarify, 3CX does indeed install Debian Security Updates, however, this is only if the "Automatic Updates" option is enabled in the "Management Console >> Updates section".

 

[SteveITS]

just to clarify, 3CX does indeed install Debian Security Updates, however, this is only if the "Automatic Updates" option is enabled in the "Management Console >> Updates section".

Hi Chris, do you mean to say that they aren't installed at all when 3CX updates are installed manually?
Or that with automatic updates off, security updates are only installed when 3CX updates are installed?
Or that if automatic updates are on, that security updates are installed weekly even if there is no pending 3CX update?


(I am also curious because I know there are other things that are updated when updates are off, like SIP trunks)

 

[ChrisC_3CX]

This

"Hi Chris, do you mean to say that they aren't installed at all when 3CX updates are installed manually?"

...and this

"Or that if automatic updates are on, that security updates are installed weekly even if there is no pending 3CX update?"

 

[SteveITS]

Interesting. That's an important bit of knowledge. We've been installing 3CX updates manually just to have more fine grained control. So in order to get security updates we should change to automatic and let it self update, then change back.

 

[ChrisC_3CX]

Yes you could indeed do that.

 

[cobaltit]

Yes you could indeed do that.

Additional clarification would be great. We don't ever have automatic updates turned on. So if 3CX is not doing system/security updates when doing the 3CX updates manually, we either need a separate automatic update option for those, a setting for delayed automatic updates, preferably configurable, or we need to know the update cycle (daily, weekly, etc). I don't really want to go toggling automatic updates on and off but that seems to be the only option until we can get more granularity in the updates.

 

[SteveITS]

setting for delayed automatic updates, preferably configurable

That would be nice, like Windows 10 used to have a "defer __ days" type of window. Or Plesk has (had?) specific release tiers they control...early adopter, general, late adopter. Set ourselves on early and clients on general or late. I did a quick search through the Ideas forum and didn't find this suggestion.

Managing checkboxes manually is workable for a small number of PBXs but does not scale.

 

[SteveITS]

I started new thread https://www.3cx.com/community/threa...-only-installed-via-automatic-updating.79651/