No access to Web Management Console since last update

[levbe]

Since the last update (SP3) on a Debian installation, i can't access the Management Console.
The web login page says : Server Error.

When i try to restart the MC service, i get the following error :

Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.

See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
root@minipc01-3cx:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Tue 2018-02-13 18:59:19 CET; 57s ago
  Process: 962 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 960 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 935 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, s
ignal=ILL)
 Main PID: 935 (code=dumped, signal=ILL)

Does it sounds familiar to someone ?

 

[levbe]

In addition to this, i have the following error :

fév 14 10:23:41 minipc01-3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.

fév 14 10:23:41 minipc01-3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'core-dump'.

fév 14 10:23:46 minipc01-3cx kernel: iptables[DOS]: IN=enp3s0 OUT= MAC=33:33:00:00:00:01:54:b8:0a:cb:de:20:86:dd SRC=fe80:0000:0000:0000:56b8:0aff:fecb:de20 DST=ff

02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337

 

[YiannisH_3CX]

Hello @levbe

Please check your inbox as i have sent you a p.m. with what we will need to troubleshoot your issue

 

[emo89]

Hi, I had the same issue by my system.
I tried to reinstall 3CX and it works for maybe 30 minutes, after that it logged me out of my system and I can´t login again and get the same Error.
Restart of MC01 was not possible.
After I installed 3CX on Windows, because I can´t work without the PBX.
If you had an resolution, please tell and I want to install it again on my Shuttle PC.

 

[levbe]

The same here...
I've reinstalled from the ISO image, reconfigured 3cx, got access to the web console and got a "502 bad gateway" when creating extensions.
3CXPhoneSystemMC01 failed.
As soon as i get any answer, i re-post it here, of course.

 

[YiannisH_3CX]

If you had an resolution, please tell and I want to install it again on my Shuttle PC.

Can you share the model of your shuttle pc so we can look into that?

 

[emo89]

Can you share the model of your shuttle pc so we can look into that?

Shuttle NC03U with 4gb ram and 120gb ssd

 

[YiannisH_3CX]

@emo89

Can you please try running the following commend and let me know of the outcome?

sudo -u phonesystem /usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot

 

[Ramped]

Exactly the same thing here after updating earlier today.

Things worked ok for a while but the console become inaccessible after a while.
Additionally, We received an email notification;
"Following service(s) were stopped recently:
3CXPhoneSystemMC01 (IP Address)"

When logging into the web interface we now get "Server Error".

So far have tried the following;
A) Rebooted the NUC (This did bring things up a couple of times, but nothing now)

B) Ran the following via SSH: "systemctl start 3CXPhoneSystemMC01"
Which returned: "Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details."

C) Ran the following via SSH: "systemctl status 3CX*"
All looked good and returned "active (running)" except for the following:
3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Fri 2018-02-16 16:11:53 AWST; 1min 46s ago
Process: 935 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
Process: 932 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Process: 915 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, signal=ILL)
Main PID: 915 (code=dumped, signal=ILL)

Feb 16 16:11:50 VOIPServer systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=dumped, status=4/ILL
Feb 16 16:11:53 VOIPServer systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'core-dump'.

D) Out of curiosity, ran the command above as per YiannisH_3CX's comment, which returned: "Illegal instruction"


3CX Is Debian/Linux Rrunning On-Prem
Hardware is a "Leader (Australia) NUC i5".

Phones are still operational, However, access to the Management Console is not available.
Sorry to Hijack the thread!

 

[YiannisH_3CX]

@Ramped

Please check your inbox as i have sent you a p.m. with what we need to further invastigate

 

[emo89]

@emo89

Can you please try running the following commend and let me know of the outcome?

sudo -u phonesystem /usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot

Hi,
thanks for the command, but I´ve running 3CX in WIndows for the Moment.
I wait until the user ramped can tell more if it will run by him when you help him, because the Computer is running in a normal business, I can´t try out at the time, because when it doesn´t work I´ve get problems with my boss.

 

[levbe]

@Ramped : thats exactly my issue also. In my case, it might be a hardware problem but as i can see, you're not using the same hardware than me at all so...

After discussion with Yiannis, i will try a new install on a different hardware, but if you get your issue solved, please come back here with some informations, it could be useful.

 

[emo89]

@emo89

Can you please try running the following commend and let me know of the outcome?

sudo -u phonesystem /usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot

Hi, I‘ve tried your commend, but the same result like by ramped „illegal instruction“

 

[MatthewHood]

Since the last update (SP3) on a Debian installation, i can't access the Management Console.
The web login page says : Server Error.

When i try to restart the MC service, i get the following error :

Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.

See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
root@minipc01-3cx:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Tue 2018-02-13 18:59:19 CET; 57s ago
  Process: 962 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 960 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 935 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, s
ignal=ILL)
 Main PID: 935 (code=dumped, signal=ILL)

Does it sounds familiar to someone ?

We have the same issue with 3CX Pro on Debian 9 on a VM. we get the same errors...

 

[Ramped]

I have replied to @YiannisH_3CX & will wait for a response. As soon as I have an update, I will pass on any relevant feedback.

Our phones are all still working, however no Management interface.

Just so that others can see what I am seeing, I have attached a screen snip of the Managment console login page.

 

[levbe]

Exactly the same !

Have you noticed when the error occurs ? Maybe it's not important but in my case, Management Consol fails only when i create new extensions.

After some tests by Yiannis and R&D team of 3CX on my system (through ssh), they did not found anything, and suspect hardware issue. However, i passed the whole week-end to test hardware (deep memtest, cpu stress tests during hours and hours) and nothing.

I think 3CX should now really consider this issue as due to SP3 and investigate on their side !

I already lost one week by testing and waiting, and beeing without a functionnal PBX.

 

[Ramped]

@levbe

Funnily enough, I was editing/creating an extension when it stopped working!

I am still waiting on a reply from @YiannisH_3CX but would agree that if they come up with the same thing, there maybe a bigger issue.. especially considering the system has been working 110% up until the update!

 

[emo89]

I tested it on 2 identical computers (same computer, same memory, same ssd). Both have the same problem.

 

[maumee]

I ma having the exact same issue as noted here. I was editing an extension, and then the nginx / server error issue arose.

Was installed fresh 2-3 months ago from a 15.5 3CX Distro / ISO. It has been on auto-upgrades every Sunday.

Also running on Shuttle NC03U with 128GB SSD. I have remote access to this box via web and ssh. I cannot login via the web (because of the noted issue).

Advice to get this going?

 

[manfer]

Out of curiosity and as I had a test installation on a virtual machine I had decided to start it and test.

And same result. Getting Server error.

After digging to find my forgotten credentials, once I was sure I was typing correct credentials and getting "Server error" message I started trying to debug the issue.

In linux to debug issues you can read logs.

The first candidate is obviously nginx logs. Curiously access.log is logging nothing at all. I suppose it is not configured. And I got nothing useful from error.log that only was showing an error on checking the letsencrypt certificate, which is expired (long time since last run of the test machine).

I restarted nginx serveral times just in case it was not running.

And started looking more logs. There are more logs to inspect. One of them is syslog were I could read:

systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument

that message doesn't sound very good, does it?

So I stopped nginx just to see /run/nginx.pid disappears correctly. And started it again to see the file is created properly. I read the file to found out it actually contains a PID number and I use ps command to actually confirm it is nginx running with that process ID.

So, what's wrong. Well, I didn't have a clue at that point but I asked google about that error message on syslog. And found a known bug:
https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1581864

So I did the suggested workaround

cd /etc/systemd/system
mkdir nginx.service.d
printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > ./nginx.service.d/override.conf
systemctl daemon-reload

And 3CX still didn't work but the error on syslog disappeared.

So decided to restart the machine.

Management console accesible again.

Reason of the problem? I suppose it is not really v15.5 SP3 as my test 3CX was not updated, but unattended updates bringing that bug into nginx maybe. Could be possible too, that this issue, though same symptom, is not exactly the same as yours.