Failed to start 3CX PhoneSystem 01 Management Console

[OlegHo]

Hello everyone, I'm having a disaster ... with 3CXManagementConsole
Debian system, everything worked fine until today, version 3CX 16.0.9
when trying to enter the web interface, it writes server error
here is the log

root@centos-2gb-fsn1-4:~# systemctl start 3CXPhoneSystemMC01
root@centos-2gb-fsn1-4:~# systemctl status 3CXPhoneSystemMC01
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Tue 2021-07-13 02:27:08 MSK; 1min 2s ago
  Process: 2840 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 2838 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 2825 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=exited, status=0/SUCCESS)
 Main PID: 2825 (code=exited, status=0/SUCCESS)

Jul 13 02:27:07 centos-2gb-fsn1-4 systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Jul 13 02:27:08 centos-2gb-fsn1-4 systemd[1]: Started 3CX PhoneSystem 01 Management Console.
root@centos-2gb-fsn1-4:~# service 3CXPhoneSystemMC01 start
Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered to the control process.
See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
root@centos-2gb-fsn1-4:~# ^C
root@centos-2gb-fsn1-4:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Tue 2021-07-13 02:29:41 MSK; 42s ago
  Process: 2974 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 2971 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 2928 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=killed, signal=ABRT)
 Main PID: 2928 (code=killed, signal=ABRT)

Jul 13 02:29:36 centos-2gb-fsn1-4 systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Jul 13 02:29:41 centos-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Jul 13 02:29:41 centos-2gb-fsn1-4 systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
Jul 13 02:29:41 centos-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Jul 13 02:29:41 centos-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.
root@centos-2gb-fsn1-4:~#

help me please

 

[Evolute IT]

Run "df -uh" and see if you still have hard disk space. It could be full, specially if you forgot Verbose mode.

 

[OlegHo]

they said:
df -uh
df: invalid option -- 'u'
Try 'df --help' for more information.

I have 20Gb free space

 

[jcostlow]

From your output I see the system hostname is centos-2gb-fsn1-4 are you running 3CX manually installed on CentOS or is that what you named the system?

 

[VasilisV_3CX]

Hi @OlegHo,

Installing via our 3CX ISO here, will install Debian 9 for our production V16. I'd suggest creating a new VM and starting fresh with our ISO.

 

[OlegHo]

Hi, a few techical details about our problems:

"We have 3cxpbx 16.0.8.9 on Debian 9.11 and at Jul 12 14:28:48 UTC 3CXPhoneSystemMC01 service unexpectedly crashes.

from `/var/log/daemon.log`

Jul 12 14:28:48 debian-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Jul 12 14:28:48 debian-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Jul 12 14:28:48 debian-2gb-fsn1-4 systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.

After `systemctl 3CXPhoneSystemMC01.service restart` service also crashes at startup. In `systemctl status 3CXPhoneSystemMC01.service` we get exception

...
Unhandled exception. System.IO.IOException: Invalid argument
at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
at System.IO.FileStream.Seek(Int64 offset, SeekOrigin origin)
at FreeImageAPI.IO.FreeImageStreamIO.streamSeek(fi_handle handle, Int32 offset, SeekOrigin origin)
at FreeImageAPI.FreeImage.GetFileTypeFromHandle(FreeImageIO& io, fi_handle handle, Int32 size)
at FreeImageAPI.FreeImage.OpenMultiBitmapFromStream(Stream stream, FREE_IMAGE_FORMAT& format, FREE_IMAGE_LOAD_FLAGS flags)
Fatal error. Internal CLR error. (0x80131506)
...

We tried to restore a several day VM image and get same error on this service. Also we restore VM image to another location with different IP, but no luck.

We use strace for service startup command `/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot` for debuging, but we did not find any request accesses to a non-existent image on the disk, it was probably taken from a pipe or another stream.

Before thread crashes it check non-existent file, but it's not a problem.
stat("/usr/lib/3cxpbx/FreeImage.Standard.pdb", 0x7fe57cff6170) = -1 ENOENT (No such file or directory)

After remark in `/usr/lib/3cxpbx/3CXManagementConsole.deps.json` section with FreeImage.Standard/4.3.8-3cx service start normaly, but when send images in client app's it return error.

At Jul 13 ~ 02:00:00 UTC error stopped appearing, even if we restored the image where it appeared earlier.

We have several 3CX PBX on the same cloud provider, on one of them this problem also appeared in Juk 12 14:10:47 UTC, but after restarting the VM, problem disappeared. Others 3CX PBX on this cloud provider did not have these problems at all."


- One our PBX from several unexpectedly crashes. All VM images does not work too.
- We are writing message on this forum.
- The error mysteriously disappears, PBX works fine. Previously not working VM images works without any problems.

What was that?

 

[jcostlow]

There could be several things that cause that issue but if you restored a VM image it may come back.

If it happens again you may be best to follow the advice from @VasilisV_3CX and re-install from scratch using the iso and restoring a 3CX backup.