Can't access Mangagement Console - Server Error

[Yellow_Flash]

Hi,

I'm having issues with our Debian(v16 iso downloaded from 3cx wesbite) Install.
I had logged into the management console a while back and one of the services wasn't running, I believe it was database server.
I tried to restart it and from that point on I'm not able to login to the Management Console.
When I try to login, it gives me "Server Error"

I am attaching the messages I received when I tried to run the following commands.

service 3CXPhoneSystemMIC01 status
• 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-11-08 04:56:32 GMT; 25min ago
Process: 4140 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
Process. 4134 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Process. 4113 ExecStart=/usr/lib/3cxpbx/3cxmanagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=exited, status=137)
Main PID: 4113 (code=exited, status=137)
Nov 08 04:56:31 3cx systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Nov 08 04:56:32 3cx systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=exited , status=137/n/a
Nov 08 04:56:32 3cx systemd[1]: Failed to start 3CX Phonesystem 01 Management Console.
Nov 08 04:56:32 3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Nov 08 04:56:32 3cx systemd[11: 3CXPhoneSystemMC01.service: Failed with result 'exit-code'.


service 3CXPhoneSystemMC01 start
Job for 3CXPhoneSystemMC01.service failed because the control process exited with error code.
See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.


Has someone encounterd the same issue, can someone let me know what can be done to resolve the issue?


Thank you.

 

[Yellow_Flash]

I haven't tried "systemctl start 3CXPhoneSystemMC01.service", I will do it next.
I'm not sure whether it will fix the issue but will try it anyway.

But I wanted to know what could be the reason for the issue and what could be done to fix it.

 

[Yellow_Flash]

When running the service 3CX* status comand, I see that...






3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-11-08 05:22:31 GMT; 4h 23min ago
Process: 8630 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
Process: 8629 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Process: 8624 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=exited, status=137)
Main PID: 8624 (code=exited, status=137)

Nov 08 05:22:30 3cx systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Nov 08 05:22:31 3cx systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=exited, status=137/n/a
Nov 08 05:22:31 3cx systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
Nov 08 05:22:31 3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Nov 08 05:22:31 3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'exit-code'.




3CXQueueManager01.service - 3CX PhoneSystem 01 Queue Manager Server
Loaded: loaded (/lib/systemd/system/3CXQueueManager01.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-11-08 04:57:02 GMT; 4h 49min ago
Process: 4247 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXQueueManager01.pid (code=exited, status=0/SUCCESS)
Process: 4240 ExecStart=/usr/lib/3cxpbx/TcxQMsrv -p /var/run/phonesystem/3CXQueueManager01.pid /var/lib/3cxpbx/Bin/config.json (code=exited, status=137)
Main PID: 4240 (code=exited, status=137)

Nov 08 04:57:02 3cx systemd[1]: Starting 3CX PhoneSystem 01 Queue Manager Server...
Nov 08 04:57:02 3cx systemd[1]: 3CXQueueManager01.service: Main process exited, code=exited, status=137/n/a
Nov 08 04:57:02 3cx systemd[1]: Failed to start 3CX PhoneSystem 01 Queue Manager Server.
Nov 08 04:57:02 3cx systemd[1]: 3CXQueueManager01.service: Unit entered failed state.
Nov 08 04:57:02 3cx systemd[1]: 3CXQueueManager01.service: Failed with result 'exit-code'.





3CXSystemService01.service - 3CX PhoneSystem 01 System Server
Loaded: loaded (/lib/systemd/system/3CXSystemService01.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-11-08 04:56:32 GMT; 4h 49min ago
Process: 4136 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXSystemService01.pid (code=exited, status=0/SUCCESS)
Process: 4116 ExecStart=/usr/lib/3cxpbx/3cxSystemService -p /var/run/phonesystem/3CXSystemService01.pid /var/lib/3cxpbx/Bin (code=exited, status=137)
Main PID: 4116 (code=exited, status=137)

Nov 08 04:56:31 3cx systemd[1]: Starting 3CX PhoneSystem 01 System Server...
Nov 08 04:56:32 3cx systemd[1]: 3CXSystemService01.service: Main process exited, code=exited, status=137/n/a
Nov 08 04:56:32 3cx systemd[1]: Failed to start 3CX PhoneSystem 01 System Server.
Nov 08 04:56:32 3cx systemd[1]: 3CXSystemService01.service: Unit entered failed state.
Nov 08 04:56:32 3cx systemd[1]: 3CXSystemService01.service: Failed with result 'exit-code'.





3CXCallFlow01.service - 3CX PhoneSystem 01 CallFlow Server
Loaded: loaded (/lib/systemd/system/3CXCallFlow01.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-11-08 04:57:02 GMT; 4h 49min ago
Process: 4254 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXCallFlow01.pid (code=exited, status=0/SUCCESS)
Process: 4241 ExecStart=/usr/lib/3cxpbx/3CXCallFlow -p /var/run/phonesystem/3CXCallFlow01.pid /var/lib/3cxpbx/Bin/config.json (code=exited, status=137)
Main PID: 4241 (code=exited, status=137)

Nov 08 04:57:02 3cx systemd[1]: Starting 3CX PhoneSystem 01 CallFlow Server...
Nov 08 04:57:02 3cx systemd[1]: 3CXCallFlow01.service: Main process exited, code=exited, status=137/n/a
Nov 08 04:57:02 3cx systemd[1]: Failed to start 3CX PhoneSystem 01 CallFlow Server.
Nov 08 04:57:02 3cx systemd[1]: 3CXCallFlow01.service: Unit entered failed state.
Nov 08 04:57:02 3cx systemd[1]: 3CXCallFlow01.service: Failed with result 'exit-code'.

 

[JohnS_3CX]

Hi,

For starters, type sudo 3CXStopServices && 3CXStartServices

This will stop any running services and then restart them

 

[Yellow_Flash]

Running the command journalctl -xe, I get


Nov 08 09:41:01 3cx liblogging-stdlog[425]: action 'action 11' resumed (module 'builtinmfile') [v8.24.0 try http://www.rsyslog.com/e
/2359 ]
Nov 08 09:41:01 3cx liblogging-stdlog[425]: action 'action 11' suspended, next retry is Fri Nov 8 09:41:31 2019 [v8.24.0 try
http://www.rsyslog.com/e/2007 ]
Nov 08 09:42:01 3cx CRON[20221]: (CRON) error (create tmpfile)

 

[Yellow_Flash]

Hi,

For starters, type sudo 3CXStopServices && 3CXStartServices

This will stop any running services and then restart them

Thank you.
I tried sudo 3CXStopServices, but got the error

Failed to disable unit: File /etc/systemd/system/3CXSystemService01.service: Read-only file system

 

[JohnS_3CX]

do you have sufficient privileges?

 

[Yellow_Flash]

do you have sufficient privileges?

Yes, I'm running the command as root.

 

[Yellow_Flash]

Do I need to use sudo, if the user is root?

 

[JohnS_3CX]

So what is the ISO installed on? a VM, a physical machine, a cloud service?

Was the machine rebooted at least once?

 

[JohnS_3CX]

avoid using sudo then if you are already root

 

[Yellow_Flash]

So what is the ISO installed on? a VM, a physical machine, a cloud service?

Was the machine rebooted at least once?

I believe I have done it before as this install was made quite a while back and it has been running properly, but I could do it now if that is necessary.

 

[JohnS_3CX]

Since you have problematic behavior of the services then a reboot will not hurt. Try rebooting the machine and give it a couple of minutes before logging into the management console

 

[Yellow_Flash]

Since you have problematic behavior of the services then a reboot will not hurt. Try rebooting the machine and give it a couple of minutes before logging into the management console

That has solved the issue.
A reboot was all it needed.
Thank you @JohnS_3CX
But when I checked the reboot history, there was a reboot done on October 1.

Any ideas on why the issue happened?

 

[JohnS_3CX]

None at all in this case I'm afraid, but can you share the specs of the machine?

It could be a low memory issue causing services to go down. You could set up a swap partition to avoid this

 

[Yellow_Flash]

None at all in this case I'm afraid, but can you share the specs of the machine?

It could be a low memory issue causing services to go down. You could set up a swap partition to avoid this

The spec is as follows.
2 CPU , 4 GB RAM
60 GB space
Do we have to increase? Or is this enough?

VMXNET adapater is being used on the machine, and they say VMTools has to be installed which we haven't. It might be a reason.

 

[JohnS_3CX]

Oh not necessarily, i have ran successfully without the VMtools on test PBX systems running Debian.

the ram should be enough but it may still ran out when you least expect it, you can install a swap file by running the following as root: apt install net-tools dphys-swapfile

 

[Yellow_Flash]

Oh not necessarily, i have ran successfully without the VMtools on test PBX systems running Debian.

the ram should be enough but it may still ran out when you least expect it, you can install a swap file by running the following as root: apt install net-tools dphys-swapfile

I just had a check and I see that we do have a swap of 4GB. So I believe it is good.

 

[Yellow_Flash]

Oh not necessarily, i have ran successfully without the VMtools on test PBX systems running Debian.

Thanks, so that is one less thing to be worried about.

Since low memory isn't the issue, what else can cause the server to behave like this?
What all do I have to be vary off?

 

[JohnS_3CX]

Do not install other services or packages on this machine, keep it dedicated to the PBX. This includes management software that may potentially update dependencies/libraries that break compatibility with your system. Avoid changing OS wide settings after the installation is complete (ie. like Locale Settings). Avoid running apt upgrade commands manually, the PBX will take care of it own dependencies via the management console if needed will update them if not it will keep the ones that have been tested to work with 3CX.

Other than that, Debian 9 is very stable and it should not cause you any issues. Not sure what happened in your specific case, but monitor the machine for a week or so and see how it goes.