Solved 3CX Stopped Working

[Tahir]

3CX was working fine, I performed a backup and then it presented Bad gateway error on the console, and now just shows server error when attempting to logon.

SSH shows the following when running the command: service 3CX* status, any help to resolve this would be appreciated as we currently have no working phone lines as a result.


root@:~# service 3CX* status
● 3CXQueueManager01.service - 3CX PhoneSystem 01 Queue Manager Server
Loaded: loaded (/lib/systemd/system/3CXQueueManager01.service; enabled)
Active: failed (Result: core-dump) since Sat 2018-11-24 13:26:09 CET; 9min ago
Process: 1053 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXQueueManager01.pid (code=exited, status=0/SUCCESS)
Process: 1021 ExecStart=/usr/lib/3cxpbx/TcxQMsrv -p /var/run/phonesystem/3CXQueueManager01.pid /var/lib/3cxpbx/Instance1/Bin/config.json (code=dumped, signal=ABRT)
Main PID: 1021 (code=dumped, signal=ABRT)

Starting 3CX PhoneSystem 01 Queue Manager Server...
3CXQueueManager01.service: main process exited, code=dumped, status=6/ABRT
Failed to start 3CX PhoneSystem 01 Queue Manager Server.
Unit 3CXQueueManager01.service entered failed state.

● 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service
Loaded: loaded (/lib/systemd/system/3CXCfgServ01.service; enabled)
Active: inactive (dead) since Sat 2018-11-24 13:29:45 CET; 6min ago
Process: 1482 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXCfgServ01.pid (code=exited, status=0/SUCCESS)
Process: 1461 ExecStart=/usr/sbin/3CXSLDBServ -n -p /var/run/phonesystem/3CXCfgServ01.pid -o /var/lib/3cxpbx/Instance1/Bin/Objects.cls -c /var/lib/3cxpbx/Instance1/Bin/3CXPhoneSystem.ini (code=exited, status=0/SUCCESS)
Main PID: 1461 (code=exited, status=0/SUCCESS)

Started 3CX PhoneSystem Configuration Service.

● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled)
Active: failed (Result: timeout) since Sat 2018-11-24 13:31:45 CET; 4min 15s ago
Process: 1484 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=killed, signal=INT)
Main PID: 1484 (code=killed, signal=INT)

Starting 3CX PhoneSystem 01 Management Console...
3CXPhoneSystemMC01.service start operation timed out. Terminating.
Failed to start 3CX PhoneSystem 01 Management Console.
Unit 3CXPhoneSystemMC01.service entered failed state.


Anyone seen this before and know how to fix?

 

[eddv123]

I guess by the inclusion of "root" in your ticket you are running on Linux, please confirm.

Either way you should check that the 3CX services are started on the device:
https://www.techrepublic.com/article/how-to-start-stop-and-restart-services-in-linux/

Follow this link also for help:
https://www.3cx.com/community/threads/502-bad-gateway.48607/

 

[Tahir]

Yes its Linux and I have paid £75 for a premium ticket expecting a quick response, sadly and most disappointingly at 2 hours 18 mins in, it still shows 'Being Processed'.

 

[eddv123]

I cannot confirm what the 3CX support teams SLA period is neither how quick they are at the weekends as it has been a while since I have dealt directly.

Please look at the two suggestions I have included above and confirm back

 

[Tahir]

Thanks for those, neither really help as one advises a reinstall and the other advises starting services which I have done but they are throwing an error which I cannot fix. I am not a Linux expert so not too comfortable with my way around it and not much online to assist with this error. Strange as nothing has even changed on the 3CX install.

 

[eddv123]

Have you simply tried Linux updates:

sudo apt-get update
sudo apt-get upgrade

Either way, for a quick fix a re-install maybe an option, if no one is responding to your ticket.

I am not 100% as it was quite a while ago since I saw such an error but if I recollect I am sure local IP of the Linux machine vs Local IP of the PBX clashing had something to do with it.

 

[Tahir]

Yes tried those and neither worked either.


apt-get update
Hit http://downloads.3cx.com InRelease
Ign http://ftp.debian.org jessie InRelease
Hit http://ftp.debian.org jessie Release.gpg
Hit http://ftp.debian.org jessie Release
Hit http://security.debian.org jessie/updates InRelease
Get:1 http://downloads.3cx.com Packages [2,003 B]
Get:2 http://ftp.debian.org jessie/main amd64 Packages [6,818 kB]
Get:3 http://security.debian.org jessie/updates/main amd64 Packages [624 kB]
Err http://security.debian.org jessie/updates/main amd64 Packages

Ign http://downloads.3cx.com Translation-en_US
Ign http://downloads.3cx.com Translation-en
Ign http://ftp.debian.org jessie/main Translation-en
Err http://ftp.debian.org jessie/main amd64 Packages
404 Not Found [IP: 130.89.148.12 80]
Get:4 http://security.debian.org jessie/updates/main amd64 Packages [624 kB]
Err http://security.debian.org jessie/updates/main amd64 Packages

0% [Waiting for headers]^[[C
Ign http://security.debian.org jessie/updates/main Translation-en
Err http://security.debian.org jessie/updates/main amd64 Packages
404 Not Found [IP: 151.101.120.204 80]
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie/main/binary-amd64/Packages 404 Not Found [IP: 130.89.148.12 80]

W: Failed to fetch http://security.debian.org/dists/jessie/updates/main/binary-amd64/Packages 404 Not Found [IP: 151.101.120.204 80]

apt-get upgrade
Bus errorackage lists... 0%

Was hoping to avoid a re-install to be honest.

 

[eddv123]

How did you install this system, via ISO or Linux first and then 3CX on-top (which is the old school way of doing this) ?

I see a lot of "jessie" in your logs here (or Debian 8 as it was known) https://www.debian.org/releases/jessie/

Debian 9 (stretch) is the supported version for 3CX currently:
https://www.3cx.com/docs/manual/

Also I found one of the 3CX guys posted a Linux cheet sheet, this may assist:
https://www.3cx.com/community/threads/linux-commands.51810/

 

[Tahir]

I used the 3CX tool to install it onto an OVH cloud hosted VPS.

 

[Tahir]

I dont think its just the cheat sheet I need, I need to understand the output of the log to see where the 3CX system is falling over so I can fix it, then I could do with the cheat sheet

 

[eddv123]

Please check your current version with the command(s) either:

cat /etc/debian_version

or: /etc/os-release

 

[Tahir]

Says 8.7

 

[Marari]

sudo apt-get update
sudo apt-get full-upgrade

This will update your debian to 9.x

 

[Tahir]

How much space do I need on the box?

 

[Marari]

How much space do I need on the box?

Not much. 200-300 MB is MORE than enough to do the upgrade.

 

[Tahir]

I think the 3CX backup I took ate all the space on the box, a friend of mine has been helping. So removed all that and rebooting the VPS to see if it works, if so I owe him the £75 instead of 3CX support who actually have not helped. Folks in the forum here have been way more help.

 

[Tahir]

Thanks for everyone who tried to help me out. glad I got away without a re-install.

 

[Tahir]

Just to add it now shows Debian 8.11 is that the latest supported version?

 

[eddv123]

sudo apt-get update
sudo apt-get full-upgrade

This was already suggested - worth reading the string before responding.

@Tahir glad you got it resolved finally - but I guess this mean 3CX paid support doesn't cover weekends.

I think it would be useful someone confirming this status for other re-sellers who land in this position over a weekend and try and get support.

 

[Tahir]

This was already suggested - worth reading the string before responding.

@Tahir glad you got it resolved finally - but I guess this mean 3CX paid support doesn't cover weekends.

I think it would be useful someone confirming this status for other re-sellers who land in this position over a weekend and try and get support.

Thanks Eddv123, however sometimes it pays not to be too eager. After upgrading 3CX to the latest stable version it already outlined the following notice and option to upgrade directly through 3CX:

'3CX detected a new Debian OS Upgrade - Debian 9. Click Upgrade to proceed'

And it also states the following:

'This action will upgrade your Debian OS to the latest version.

Important: OS Debian update scripts will only work if you installed 3CX using the 3CX ISO and is tested against the following cloud hosting providers: Google Cloud, Amazon, Azure and OVH.

Before you perform this update, take a backup of your 3CX Configuration and save it somewhere safe and accessible outside of the instance.

This update script will stop all 3CX Services, uninstall 3CX keeping all the data in place, upgrade Debian OS and install the latest 3CX Version. The update should take approximately 20 minutes.'

So my advice would be to use the 3CX upgrade tool rather than try to do it via the command line unless you really have too but also take the above notice into consideration.

With regards to the support, yes I agree it would be helpful if 3CX outlined the supported hours via ticket and phone etc within the partner portal regardless of partner status. I just hope they refund the support ticket I opened which has now been resolved with the help of the community and some of my friends who assisted me directly.