Services Failing after less than 24h Uptime

[imCalebM]

I have a new install hosted in Azure for which several services are regularly failing. Rebooting the VM clears the issues and the services come up OK, but less than 24h later the same 2 services fail. It first happened after initially installing 3CX, I went home for the day and found the next morning the services had failed.
I have hosted our internal 3CX in Azure for over 6 months now and have never had any issues like this and am a bit puzzled.

• Standard Annual 18.0 (Build 461)
• Debian 10.11
• Hosted in an Azure VM

The failing services are 3CXGatewayService.service and 3CXPhoneSystemMC01.service.

I have found some other posts of similar issues but they are one-offs rather than regular occurrences.

Here is the list of the 3CX services including the failed ones.

~$ systemctl list-units --type=service -all
  UNIT                                                        LOAD      ACTIVE   SUB     DESCRIPTION
  3CXAudioProvider01.service                                  loaded    active   running 3CX PhoneSystem 01 Audio Provider Server
  3CXCallFlow01.service                                       loaded    active   running 3CX PhoneSystem 01 CallFlow Server
  3CXCfgServ01.service                                        loaded    active   running 3CX PhoneSystem Configuration Service
  3CXEventNotificationManager.service                         loaded    active   running 3CX Event Notification Manager
● 3CXGatewayService.service                                   loaded    failed   failed  3CX Gateway Service
  3CXIVR01.service                                            loaded    active   running 3CX PhoneSystem 01 IVR Server
  3CXMediaServer.service                                      loaded    active   running 3CX PhoneSystem Media Server
  3CXPhoneSystem01.service                                    loaded    active   running 3CX PhoneSystem 01 SIP Server
● 3CXPhoneSystemMC01.service                                  loaded    failed   failed  3CX PhoneSystem 01 Management Console
  3CXQueueManager01.service                                   loaded    active   running 3CX PhoneSystem 01 Queue Manager Server
  3CXSystemService01.service                                  loaded    active   running 3CX PhoneSystem 01 System Server

Here is my output for systemctl status 3CXPhoneSystemMC01.service

~$ systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Mon 2022-05-09 18:38:36 NZST; 14h ago
  Process: 7673 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (
  Process: 15005 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 15013 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
 Main PID: 7673 (code=killed, signal=KILL)

May 09 16:39:27 UCC-3CX-VM sudo[12179]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 09 16:39:27 UCC-3CX-VM sudo[12179]: pam_unix(sudo:session): session closed for user root
May 09 16:39:41 UCC-3CX-VM sudo[12189]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/sbin/tcpdump -D
May 09 16:39:41 UCC-3CX-VM sudo[12189]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 09 16:39:41 UCC-3CX-VM sudo[12189]: pam_unix(sudo:session): session closed for user root
May 09 16:44:50 UCC-3CX-VM sudo[12339]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/sbin/tcpdump -D
May 09 16:44:50 UCC-3CX-VM sudo[12339]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 09 16:44:50 UCC-3CX-VM sudo[12339]: pam_unix(sudo:session): session closed for user root
May 09 18:38:35 UCC-3CX-VM systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=9/KILL
May 09 18:38:36 UCC-3CX-VM systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.

Appreciate any advice out there!

 

[xenexsystems]

Our PBXs have started randomly locking up on Debian 10 and V18 on AWS. Was stable then since doing the upgrades they intermittently lock up.

The only way we've been able to stabilise them is add more RAM

 

[imCalebM]

Our PBXs have started randomly locking up on Debian 10 and V18 on AWS. Was stable then since doing the upgrades they intermittently lock up.

The only way we've been able to stabilise them is add more RAM

Damn, I was hoping to get by with the minimum requirements as indicated here: https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/

I'll double the RAM to 2GB and see how it goes. Our internal system has 2GB of RAM too and is stable, suggesting RAM is the issue too.

 

[NickD_3CX]

Damn, I was hoping to get by with the minimum requirements as indicated here: https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/

I'll double the RAM to 2GB and see how it goes. Our internal system has 2GB of RAM too and is stable, suggesting RAM is the issue too.

Not sure how you installed 3CX, but you might want to consider taking a backup of your 3CX system, downloading it to your local PC, deleting the Azure instance and re-creating it from our Customer Portal or Marketplace.

Although I am not 100% on the details, I seem to recall there was a problem with the swap file in Azure in general where funky things were happening. On machines with less RAM, as you can understand swap file is used quite a lot.

I also remember though hearing that we found a way to work around it for deployments, hence my recommendation to re-create a new Azure instance and restore your backup of course.