Toggle sidebar Toggle sidebar

Solved Service failed to start after upate

Status
Not open for further replies.

pauly

Free User
Joined
Apr 8, 2020
Messages
17
Reaction score
5
Hi,

I got an email that said 3cx successfully updated.

Later that day, clients could not connect to chat.
I tried restarting all services from the dashboard, but that did not help.
Then I tried a reboot and a restart. and got these errors:
Any help appreciated.

Bash: 
systemctl start 3CXPhoneSystemMC01

A dependency job for 3CXPhoneSystemMC01.service failed. See 'journalctl -xe' for details.

root@ip-172-26-11-179:~# journalctl -xe

--

-- Userspace start-up required 113451578 microseconds.

Mar 22 19:22:29 ip-172-26-11-179 sudo[714]:    admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/bin/su -

Mar 22 19:22:29 ip-172-26-11-179 sudo[714]: pam_unix(sudo:session): session opened for user root by admin(uid=0)

Mar 22 19:22:29 ip-172-26-11-179 su[715]: (to root) admin on pts/0

Mar 22 19:22:29 ip-172-26-11-179 su[715]: pam_unix(su-l:session): session opened for user root by admin(uid=0)

Mar 22 19:22:42 ip-172-26-11-179 systemd[1]: Starting 3CX PhoneSystem Configuration Service...

-- Subject: A start job for unit 3CXCfgServ01.service has begun execution

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit 3CXCfgServ01.service has begun execution.

--

-- The job identifier is 234.

Mar 22 19:22:57 ip-172-26-11-179 dhclient[418]: XMT: Solicit on eth0, interval 130610ms.

Mar 22 19:24:12 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Start operation timed out. Terminating.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: State 'stop-sigterm' timed out. Killing.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Killing process 725 (3CXSLDBServ) with signal SIGKILL.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Main process exited, code=killed, status=9/KILL

-- Subject: Unit process exited

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- An ExecStart= process belonging to unit 3CXCfgServ01.service has exited.

--

-- The process' exit code is 'killed' and its exit status is 9.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Failed with result 'timeout'.

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit 3CXCfgServ01.service has entered the 'failed' state with result 'timeout'.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: Failed to start 3CX PhoneSystem Configuration Service.

-- Subject: A start job for unit 3CXCfgServ01.service has failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit 3CXCfgServ01.service has finished with a failure.

--

-- The job identifier is 234 and the job result is failed.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: Dependency failed for 3CX PhoneSystem 01 Management Console.

-- Subject: A start job for unit 3CXPhoneSystemMC01.service has failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit 3CXPhoneSystemMC01.service has finished with a failure.

--

-- The job identifier is 233 and the job result is dependency.

Mar 22 19:24:27 ip-172-26-11-179 systemd[1]: 3CXPhoneSystemMC01.service: Job 3CXPhoneSystemMC01.service/start failed with result 'dependency'.

Mar 22 19:25:01 ip-172-26-11-179 CRON[753]: pam_unix(cron:session): session opened for user root by (uid=0)

Mar 22 19:25:01 ip-172-26-11-179 CRON[754]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Mar 22 19:25:01 ip-172-26-11-179 CRON[753]: pam_unix(cron:session): session closed for user root

lines 1562-1613/1613 (END)
 
Hi @pauly,
Are you able to SSH into the machine and type:

Code: 
sudo service 3CX* status

It should give you a list of all 3CX related services and whether they're running or not. If one is not running, start it using:

Code: 
sudo service 3CXSERVICENAME start

If all services are up and running, I'd suggest rebooting the machine. Let me know if any of that helps.
 
OK I did that, 3CXCfgServ01 had failed to start

On attempting to start got an error
I followed the instructions to output systemctl status and journalctll -xe

Bash: 
admin@ip-172-26-11-179:~$ sudo service 3CX* status

● 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service

   Loaded: loaded (/lib/systemd/system/3CXCfgServ01.service; enabled; vendor preset: enabled)

   Active: failed (Result: timeout) since Tue 2022-03-22 20:56:55 GMT; 2 days ago

  Process: 816 ExecStart=/usr/sbin/3CXSLDBServ -n -p /var/run/phonesystem/3CXCfgServ01.pid -o /var/lib/3cxpbx/Bin/Objects.cls -c /var/lib/3cxpbx/Bin/3CXPhoneSystem

  Process: 841 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXCfgServ01.pid (code=exited, status=0/SUCCESS)

 Main PID: 816 (code=killed, signal=KILL)



Mar 22 20:55:10 ip-172-26-11-179 systemd[1]: Starting 3CX PhoneSystem Configuration Service...

Mar 22 20:56:40 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Start operation timed out. Terminating.

Mar 22 20:56:55 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: State 'stop-sigterm' timed out. Killing.

Mar 22 20:56:55 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Killing process 816 (3CXSLDBServ) with signal SIGKILL.

Mar 22 20:56:55 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Main process exited, code=killed, status=9/KILL

Mar 22 20:56:55 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Failed with result 'timeout'.

Mar 22 20:56:55 ip-172-26-11-179 systemd[1]: Failed to start 3CX PhoneSystem Configuration Service.



● 3CXEventNotificationManager.service - 3CX Event Notification Manager

   Loaded: loaded (/lib/systemd/system/3CXEventNotificationManager.service; enabled; vendor preset: enabled)

   Active: active (running) since Tue 2022-03-22 20:55:08 GMT; 2 days ago

 Main PID: 553 (CloudServicesWa)

    Tasks: 11 (limit: 4699)

   Memory: 76.1M

   CGroup: /system.slice/3CXEventNotificationManager.service

           └─553 /usr/lib/3cxpbx/CloudServicesWatcher -p /var/run/phonesystem/3CXEventNotificationManager.pid -c /var/lib/3cxpbx/Bin/CloudServicesWatcher.exe.confi



Mar 22 20:55:06 ip-172-26-11-179 systemd[1]: Starting 3CX Event Notification Manager...

Mar 22 20:55:08 ip-172-26-11-179 systemd[1]: Started 3CX Event Notification Manager.



● 3CXMediaServer.service - 3CX PhoneSystem Media Server

   Loaded: loaded (/lib/systemd/system/3CXMediaServer.service; enabled; vendor preset: enabled)

   Active: active (running) since Tue 2022-03-22 20:55:06 GMT; 2 days ago

 Main PID: 554 (3CXMediaServer)

    Tasks: 19 (limit: 4699)

   Memory: 15.5M

   CGroup: /system.slice/3CXMediaServer.service

           └─554 /usr/sbin/3CXMediaServer -n -p /var/run/phonesystem/3CXMediaServer.pid -c /var/lib/3cxpbx/Bin/3CXPhoneSystem.ini -m /var/lib/3cxpbx/Bin/3cxmediase



Mar 22 20:55:06 ip-172-26-11-179 systemd[1]: Starting 3CX PhoneSystem Media Server...

Mar 22 20:55:06 ip-172-26-11-179 systemd[1]: Started 3CX PhoneSystem Media Server.

admin@ip-172-26-11-179:~$ sudo service 3CXCfgServ01 start

Job for 3CXCfgServ01.service failed because a timeout was exceeded.

See "systemctl status 3CXCfgServ01.service" and "journalctl -xe" for details.

admin@ip-172-26-11-179:~$ systemctl status 3CXCfgServ01.service

● 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service

   Loaded: loaded (/lib/systemd/system/3CXCfgServ01.service; enabled; vendor preset: enabled)

   Active: failed (Result: timeout) since Fri 2022-03-25 09:23:58 GMT; 2min 15s ago

  Process: 12458 ExecStart=/usr/sbin/3CXSLDBServ -n -p /var/run/phonesystem/3CXCfgServ01.pid -o /var/lib/3cxpbx/Bin/Objects.cls -c /var/lib/3cxpbx/Bin/3CXPhoneSystem.ini (code=kil

  Process: 12477 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXCfgServ01.pid (code=exited, status=0/SUCCESS)

 Main PID: 12458 (code=killed, signal=KILL)



Mar 25 09:22:13 ip-172-26-11-179 systemd[1]: Starting 3CX PhoneSystem Configuration Service...

Mar 25 09:23:43 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Start operation timed out. Terminating.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: State 'stop-sigterm' timed out. Killing.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Killing process 12458 (3CXSLDBServ) with signal SIGKILL.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Main process exited, code=killed, status=9/KILL

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Failed with result 'timeout'.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: Failed to start 3CX PhoneSystem Configuration Service.

lines 1-14/14 (END)

admin@ip-172-26-11-179:~$ journalctl -xe

Mar 25 09:22:13 ip-172-26-11-179 systemd[1]: Starting 3CX PhoneSystem Configuration Service...

-- Subject: A start job for unit 3CXCfgServ01.service has begun execution

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit 3CXCfgServ01.service has begun execution.

--

-- The job identifier is 1206.

Mar 25 09:23:43 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Start operation timed out. Terminating.

Mar 25 09:23:43 ip-172-26-11-179 dhclient[462]: XMT: Solicit on eth0, interval 108710ms.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: State 'stop-sigterm' timed out. Killing.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Killing process 12458 (3CXSLDBServ) with signal SIGKILL.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Main process exited, code=killed, status=9/KILL

-- Subject: Unit process exited

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- An ExecStart= process belonging to unit 3CXCfgServ01.service has exited.

--

-- The process' exit code is 'killed' and its exit status is 9.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: 3CXCfgServ01.service: Failed with result 'timeout'.

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit 3CXCfgServ01.service has entered the 'failed' state with result 'timeout'.

Mar 25 09:23:58 ip-172-26-11-179 systemd[1]: Failed to start 3CX PhoneSystem Configuration Service.

-- Subject: A start job for unit 3CXCfgServ01.service has failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit 3CXCfgServ01.service has finished with a failure.

--

-- The job identifier is 1206 and the job result is failed.



Is there a way to dump my config - extensions trunks, etc and run up a new instance?
 
Hi @pauly,

We previously had an issue where certain services were disabled, can you run the following to check:


Bash: 
systemctl is-enabled 3CXIVR01.service
systemctl is-enabled 3CXPhoneSystem01.service
systemctl is-enabled 3CXMediaServer.service
systemctl is-enabled 3CXCallFlow01.service
systemctl is-enabled 3CXFirewall.service
systemctl is-enabled 3CXAudioProvider01.service
systemctl is-enabled 3CXCfgServ01.service
systemctl is-enabled 3CXEventNotificationManager.service
systemctl is-enabled 3CXSystemService01.service
systemctl is-enabled 3CXGatewayService.service
systemctl is-enabled 3CXPhoneSystemMC01.service
systemctl is-enabled 3CXQueueManager01.service
systemctl is-enabled nginx.service

and if any are disabled, run:

Bash: 
systemctl enable 3CXIVR01.service
systemctl enable 3CXPhoneSystem01.service
systemctl enable 3CXMediaServer.service
systemctl enable 3CXCallFlow01.service
systemctl enable 3CXFirewall.service
systemctl enable 3CXAudioProvider01.service
systemctl enable 3CXCfgServ01.service
systemctl enable 3CXEventNotificationManager.service
systemctl enable 3CXSystemService01.service
systemctl enable 3CXGatewayService.service
systemctl enable 3CXPhoneSystemMC01.service
systemctl enable 3CXQueueManager01.service
systemctl enable nginx

Then give it a restart again.
 
all already enabled:

admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXIVR01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXPhoneSystem01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXMediaServer.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXCallFlow01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXFirewall.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXAudioProvider01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXCfgServ01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXEventNotificationManager.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXSystemService01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXGatewayService.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXPhoneSystemMC01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled 3CXQueueManager01.service
enabled
admin@ip-172-26-11-179:~$ systemctl is-enabled nginx.service
enabled


rebooted instance, but no success
 
I just tried re-install on a new LightSail instance.
It fell over after about 10 minutes after I had set up the first trunk and iI get the same errors as previously.
 
pauly said:
I just tried re-install on a new LightSail instance.
It fell over after about 10 minutes after I had set up the first trunk and iI get the same errors as previously.
Click to expand...
Hi!

So just to be clear, you re-installed 3CX on a new LightSail machine and for the first 10 minutes it was OK, but after 10 minutes, but after that the same thing happened again?

If this is what happened, could you please walk us through exactly how you deployed the new LightSail instance? Was it using the Marketplace image or the Deployment Wizard? Also how much RAM was allocated to the instance you created?
 
Hi
Yes I set up a new lightsail instance using the reinstall button on the portal,

1648475131278.png

Then I set up a trunk.
At that point I lost connection and when I ssh'd into the box the 3cx phone config service had failed just like in the old server.

I set it up with 4GB RAM and 2CPU
 
Hi @pauly,

Do you have a core file in /var/lib/3cxpbx?

Also, how is the machine's resource usage? Is there enough disk space? Is there any shortage of RAM etc?
 
Hi
No core file in /var/lib/3cxpbx
8% disk space used 70G free
> 3GB RAM free
 
I'll PM you shortly for more info on this.
 
An update on this, @pauly eventually deployed a new machine via the 3CX portal to Lightsail and will configure it from scratch. For now everything seems normal.
 
Many thanks to @ChrisC_3CX for spending time on this. it was much appreciated.
 
  • Like
Reactions: ChrisC_3CX and NickD_3CX
My pleasure!

Since everything seems fine now I'll be marking this as solved, but do let us know if anything comes up!
 
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,972
Messages
590,066
Members
164,888
Latest member
Tim Cook
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom