502 Bad Gateway

[JST]

Our system suddenly shows a popup with 502 Bad Gateway and it is no longer possible to logon to the web admin console. The system runs on VMware with one assigned NIC and there has been no change to the system (saw the other posts about having two NICs).

Any suggestions how we can restore web admin access?

 

[YiannisH_3CX]

Hello @JST

Please make sure that all services are running as this sounds like your management console service is down.
Please check and let us know

 

[JST]

That makes sense, but how can I do that on Debian?

 

[YiannisH_3CX]

You need SSH access. Once you have access you can check the services by typing service 3CX* status. If the management console is indeed down you can start it by typing service 3CXPhoneSystemMC01 start

 

[JST]

I am getting the following output:

root@pbx:~# service 3CX* status
● 3CXQueueManager01.service - 3CX PhoneSystem 01 Queue Manager Server
Loaded: loaded (/lib/systemd/system/3CXQueueManager01.service; enabled)
Active: active (running) since Thu 2017-04-20 07:07:06 PDT; 3h 1min ago
Main PID: 721 (TcxQMsrv)
CGroup: /system.slice/3CXQueueManager01.service
└─721 /usr/lib/3cxpbx/TcxQMsrv -p /var/run/phonesystem/3CXQueueMan...

Apr 20 07:07:06 pbx systemd[1]: Started 3CX PhoneSystem 01 Queue Manager Server.

● 3CXTunnel01.service - 3CX PhoneSystem 01 SIP/RTP Tunneling Proxy
Loaded: loaded (/lib/systemd/system/3CXTunnel01.service; enabled)
Active: active (running) since Thu 2017-04-20 07:06:57 PDT; 3h 1min ago
Main PID: 560 (3CXTunnel)
CGroup: /system.slice/3CXTunnel01.service
└─560 /usr/sbin/3CXTunnel -n -p /var/run/phonesystem/3CXTunnel01.p...

Apr 20 07:06:55 pbx systemd[1]: Starting 3CX PhoneSystem 01 SIP/RTP Tunneli.....
Apr 20 07:06:57 pbx systemd[1]: Started 3CX PhoneSystem 01 SIP/RTP Tunnelin...y.

● 3CXPhoneSystem01.service - 3CX PhoneSystem 01 SIP Server
Loaded: loaded (/lib/systemd/system/3CXPhoneSystem01.service; enabled)
Active: active (running) since Thu 2017-04-20 07:07:03 PDT; 3h 1min ago
Main PID: 700 (3CXPhoneSystem)
CGroup: /system.slice/3CXPhoneSystem01.service
└─700 /usr/sbin/3CXPhoneSystem -p /var/run/phonesystem/3CXPhoneSys...

Apr 20 07:07:03 pbx systemd[1]: Started 3CX PhoneSystem 01 SIP Server.

● 3CXIVR01.service - 3CX PhoneSystem 01 IVR Server
Loaded: loaded (/lib/systemd/system/3CXIVR01.service; enabled)
Active: active (running) since Thu 2017-04-20 07:07:03 PDT; 3h 1min ago
Main PID: 722 (3CXIVR)
CGroup: /system.slice/3CXIVR01.service
└─722 /usr/sbin/3CXIVR -n -p /var/run/phonesystem/3CXIVR01.pid -c ...

Apr 20 07:07:03 pbx systemd[1]: Started 3CX PhoneSystem 01 IVR Server.

● 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service
Loaded: loaded (/lib/systemd/system/3CXCfgServ01.service; enabled)
Active: active (running) since Thu 2017-04-20 07:07:02 PDT; 3h 1min ago
Main PID: 689 (3CXSLDBServ)
CGroup: /system.slice/3CXCfgServ01.service
└─689 /usr/sbin/3CXSLDBServ -n -p /var/run/phonesystem/3CXCfgServ0...

Apr 20 07:07:02 pbx systemd[1]: Started 3CX PhoneSystem Configuration Service.

● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled)
Active: failed (Result: core-dump) since Thu 2017-04-20 07:07:05 PDT; 3h 1min ago
Process: 776 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
Process: 774 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Process: 701 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, signal=ABRT)
Main PID: 701 (code=dumped, signal=ABRT)

Apr 20 07:07:05 pbx systemd[1]: 3CXPhoneSystemMC01.service: main process ex...RT
Apr 20 07:07:05 pbx systemd[1]: Failed to start 3CX PhoneSystem 01 Manageme...e.
Apr 20 07:07:05 pbx systemd[1]: Unit 3CXPhoneSystemMC01.service entered fai...e.

● 3CXMediaServer.service - 3CX PhoneSystem Media Server
Loaded: loaded (/lib/systemd/system/3CXMediaServer.service; enabled)
Active: active (running) since Thu 2017-04-20 07:06:57 PDT; 3h 1min ago
Main PID: 567 (3CXMediaServer)
CGroup: /system.slice/3CXMediaServer.service
└─567 /usr/sbin/3CXMediaServer -n -p /var/run/phonesystem/3CXMedia...

Apr 20 07:06:57 pbx systemd[1]: Started 3CX PhoneSystem Media Server.

● 3CXSystemService01.service - 3CX PhoneSystem 01 System Server
Loaded: loaded (/lib/systemd/system/3CXSystemService01.service; enabled)
Active: failed (Result: signal) since Thu 2017-04-20 07:07:06 PDT; 3h 1min ago
Process: 803 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXSystemService01.pid (code=exited, status=0/SUCCESS)
Process: 702 ExecStart=/usr/lib/3cxpbx/3cxSystemService -p /var/run/phonesystem/3CXSystemService01.pid /var/lib/3cxpbx/Instance1/Bin (code=killed, signal=ABRT)
Main PID: 702 (code=killed, signal=ABRT)

Apr 20 07:07:05 pbx systemd[1]: Started 3CX PhoneSystem 01 System Server.
Apr 20 07:07:06 pbx systemd[1]: 3CXSystemService01.service: main process ex...RT
Apr 20 07:07:06 pbx systemd[1]: Unit 3CXSystemService01.service entered fai...e.

● 3CXEventNotificationManager.service - 3CX Event Notification Manager
Loaded: loaded (/lib/systemd/system/3CXEventNotificationManager.service; enabled)
Active: active (running) since Thu 2017-04-20 07:07:02 PDT; 3h 1min ago
Main PID: 568 (CloudServicesWa)
CGroup: /system.slice/3CXEventNotificationManager.service
└─568 /usr/lib/3cxpbx/CloudServicesWatcher -p /var/run/phonesystem...

Apr 20 07:07:02 pbx systemd[1]: Started 3CX Event Notification Manager.
Hint: Some lines were ellipsized, use -l to show in full.


It seems that two services aren't working properly:
3CX Phone System 01 Management Console
3CX Phone System 01 System Server

A restart fails with the message shown above.

I am really puzzled how this can happen when no changes were applied to the system.

What is our best recovery option now?

 

[YiannisH_3CX]

I have sent you a p.m.

 

[Barnabis]

Hello,

I ve the same problem.

i have increased the timeout in systemd and two services didn't start. I try to reboot the server but nothing ... Below the result of the command when I try to start 3CXPhoneSystemMC01

May 28 14:19:51 sl0001 systemd[1]: Starting 3CX PhoneSystem Configuration Service...
-- Subject: Unit 3CXCfgServ01.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit 3CXCfgServ01.service has begun starting up.
May 28 14:20:16 sl0001 systemd[1]: Started 3CX PhoneSystem Configuration Service.
-- Subject: Unit 3CXCfgServ01.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit 3CXCfgServ01.service has finished starting up.
--
-- The start-up result is done.
May 28 14:20:16 sl0001 systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
-- Subject: Unit 3CXPhoneSystemMC01.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit 3CXPhoneSystemMC01.service has begun starting up.
May 28 14:22:16 sl0001 systemd[1]: 3CXPhoneSystemMC01.service start operation timed out. Terminating.
May 28 14:22:16 sl0001 systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
-- Subject: Unit 3CXPhoneSystemMC01.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit 3CXPhoneSystemMC01.service has failed.
--
-- The result is failed.
May 28 14:22:16 sl0001 systemd[1]: Unit 3CXPhoneSystemMC01.service entered failed state.

When I launch the command with straw (see above) the response is continuously :

Connecting to the PBX... Check config.json!

Can you help me ?

Best regards,

Olivier

 

[JST]

Hi Olivier,

In my case, support did a remote session to my system and found that there was an issue with license activation. Unfortunately, they were unable to resolve that and so I ended up with a re-installation. While I didn't have a configuration backup, my home only has 4 extensions and so I didn't loose too much configuration.

Lesson learned: Use auto backup on a daily basis!

So, if you have a working backup, it might be faster to do a re-installation because I don't think that support will be reading your question before Monday. If you want to wait on them, you should either way be present during the remote session (European time) or setup VPN/SSH access (that's what I did).

Good luck!

Cheers,
Jens