OVH Server stopped responding - services not starting

[gbrown100]

Hiya,

I have a hosted OVH instance installed using PBX express, still debian 9 V16. Web interface was kooky this morning, a reboot meant no response from the web interface at all. Logged in and got the below when checking the service status: Any idea why they cannot start?

myuser@ovh-custom-uk1-2018-12-11-17-24-39-jouvy:~# service 3CX* status
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vend
or preset: enabled)
Active: failed (Result: timeout) since Mon 2021-08-16 09:45:46 BST
; 2min 41s ago
Process: 1007 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/ph
onesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webmyuser
(code=killed, signal=INT)
Main PID: 1007 (code=killed, signal=INT)

Aug 16 09:43:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Starting 3C
X PhoneSystem 01 Management Console...
Aug 16 09:45:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Start operation timed out. Terminating.
Aug 16 09:45:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Fai
led to start 3CX PhoneSystem 01 Management Console.
Aug 16 09:45:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Unit entered failed state.
Aug 16 09:45:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Failed with result 'timeout'.

● 3CXCallFlow01.service - 3CX PhoneSystem 01 CallFlow Server
Loaded: loaded (/lib/systemd/system/3CXCallFlow01.service; enabled; vendor pr
eset: enabled)
Active: failed (Result: timeout) since Mon 2021-08-16 09:46:01 BST
; 2min 26s ago
Process: 1213 ExecStart=/usr/lib/3cxpbx/3CXCallFlow -p /var/run/phonesystem/3C
XCallFlow01.pid /var/lib/3cxpbx/Bin/config.json (code=killed, signal=KIL
L)
Main PID: 1213 (code=killed, signal=KILL)

Aug 16 09:44:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Starting 3C
X PhoneSystem 01 CallFlow Server...
Aug 16 09:45:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: Start operation timed out. Terminating.
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: State 'stop-final-sigterm' timed out. Killing.
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: Killing process 1213 (3CXCallFlow) with signal SIGKILL.
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: Main process exited, code=killed, status=9/KILL
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Fai
led to start 3CX PhoneSystem 01 CallFlow Server.
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: Unit entered failed state.
Aug 16 09:46:01 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
CallFlow01.service: Failed with result 'timeout'.

● 3CXSystemService01.service - 3CX PhoneSystem 01 System Server
Loaded: loaded (/lib/systemd/system/3CXSystemService01.service; enabled; vend
or preset: enabled)
Active: failed (Result: timeout) since Mon 2021-08-16 09:46:16 BST
; 2min 11s ago
Process: 1003 ExecStart=/usr/lib/3cxpbx/3cxSystemService -p /var/run/phonesyst
em/3CXSystemService01.pid /var/lib/3cxpbx/Bin (code=killed, signal=KILL)

Main PID: 1003 (code=killed, signal=KILL)

Aug 16 09:43:46 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Starting 3C
X PhoneSystem 01 System Server...
Aug 16 09:45:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: Start operation timed out. Terminating.
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: State 'stop-final-sigterm' timed out. Killing.
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: Killing process 1003 (3cxSystemServic) with signal SIGK
ILL.
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: Main process exited, code=killed, status=9/KILL
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Fai
led to start 3CX PhoneSystem 01 System Server.
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: Unit entered failed state.
Aug 16 09:46:16 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
SystemService01.service: Failed with result 'timeout'.

● 3CXMediaServer.service - 3CX PhoneSystem Media Server
Loaded: loaded (/lib/systemd/system/3CXMediaServer.service; enabled; vendor p
reset: enabled)
Active: active (running) since Mon 2021-08-16 09:42:49 BST; 5min a
go
Main PID: 776 (3CXMediaServer)
Tasks: 15 (limit: 4915)
CGroup: /system.slice/3CXMediaServer.service
└─776 /usr/sbin/3CXMediaServer -n -p /var/run/phonesystem/3CXMediaSer
ver.pid -c /var/lib/3cxpbx/Bin/3CXPhoneSystem.ini -m /var/lib/3cxpbx/Bin/3cxmedi
aserver.ini


myuser@ovh-custom-uk1-2018-12-11-17-24-39-jouvy:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vend
or preset: enabled)
Active: failed (Result: timeout) since Mon 2021-08-16 09:53:50 BST
; 1min 18s ago
Process: 3820 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/ph
onesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webmyuser
(code=killed, signal=INT)
Main PID: 3820 (code=killed, signal=INT)

Aug 16 09:51:50 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Starting 3C
X PhoneSystem 01 Management Console...
Aug 16 09:53:50 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Start operation timed out. Terminating.
Aug 16 09:53:50 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Fai
led to start 3CX PhoneSystem 01 Management Console.
Aug 16 09:53:50 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Unit entered failed state.
Aug 16 09:53:50 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: 3CX
PhoneSystemMC01.service: Failed with result 'timeout'.
myuser@ovh-custom-uk1-2018-12-11-17-24-39-jouvy:~#

 

[NickD_3CX]

Via SSH try running:

sudo /usr/sbin/3CXStartServices

 

[gbrown100]

I have all services running now but still no web interface! Could this be an NGINX problem?

debian@ovh-custom-uk1-2018-12-11-17-24-39-jouvy:~$ sudo service nginx* status
● nginx.service - A high performance web server and a reverse proxy s
erver
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
abled)
Active: failed (Result: exit-code) since Mon 2021-08-16 10:33:33 B
ST; 6min ago
Docs: man:nginx(8)

Aug 16 10:33:32 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Starting A
high performance web server and a reverse proxy server...
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy nginx[775]: nginx: [eme
rg] PEM_read_bio_DHparams("/var/lib/3cxpbx/Bin/nginx/conf/Instance1/dhparam.pem"
) failed (SSL: error:0906D06CEM routinesEM_read_bio:no start line:Expecting:
DH PARAMETERS)
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy nginx[775]: nginx: conf
iguration file /etc/nginx/nginx.conf test failed
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: ngi
nx.service: Control process exited, code=exited status=1
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: Fai
led to start A high performance web server and a reverse proxy server.
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: ngi
nx.service: Unit entered failed state.
Aug 16 10:33:33 ovh-custom-uk1-2018-12-11-17-24-39-jouvy systemd[1]: ngi
nx.service: Failed with result 'exit-code'.

 

[NickD_3CX]

I have all services running now but still no web interface! Could this be an NGINX problem?

Yes, that is very possible as well.

 

[gbrown100]

Yes, that is very possible as well.

Seems like my SSL cert is corrupt? Is there a way to force the server to re-download a 3CX issued cert?

 

[NickD_3CX]

Is there a way to force the server to re-download a 3CX issued cert?

I'll send you some instructions via PM.

 

[gbrown100]

I'll send you some instructions via PM.

Thanks Nick, that would be brilliant!

 

[NickD_3CX]

On second thought, it's not possible as you would require access to he Management Console to do what I was thinking... :/
The next best thing I can think of is:

1. Get a command line Backup: https://www.3cx.com/docs/backup-restore-command-line/
2. Download the backup file.
3. Delete the VM
4. Re-deploy a new VM using the either the Configuration Wizard from the Customer Portal or MArketplace instructions and restore your backup.

 

[gbrown100]

I have backups for the last 14 days so that's OK. Last night's failed anyway so I doubt a command line will work. I have not had to do a rebuild from the customer portal before, what's the downtime likely to be whilst the domain name get's re-pointed? Will the on site SBC need a kick?

 

[NickD_3CX]

I have backups for the last 14 days so that's OK. Last night's failed anyway so I doubt a command line will work. I have not had to do a rebuild from the customer portal before, what's the downtime likely to be whilst the domain name get's re-pointed? Will the on site SBC need a kick?

Usually the deployment via the Customer Portal takes up to 15 minutes.
Make sure that before you start doing it, that you have at least powered off the previous machine.

Also, make sure you deploy using the same key your previous installation was using (the powered off one).
DO NOT release the FQDN from your License Key.

Once you do this, you'll end up with a new V18 installation with the key you used.
Log onto that machine via SFTP, upload the backup you have to the backup folder, then via command line change the owner of the file to 'phonesystem' using the following command:

sudo chown phonesystem:phonesystem <the backup file>

Then, when you log back into the Management Console and go to "Backup and Restore", you should see the file you uploaded. Just select it and hit "Restore".
The restore process time depends on the backup, but unless there are many millions of call records and it's generally a huge system, the restore takes less than 15 minutes.