Solved Services wont restart

[TLCtech]

After a reboot of the server, Several services stopped, and none of the extensions are connecting. Im running the latest software on raspian buster.




Ive tried restarting each service, and all services from the terminal. Nothing seems to work. Any help is appreciated.

 

[TLCtech]

Update: I rebooted the machine and now I cant log in at all. Says "server error".

 

[TLCtech]

Update 2: I found and ran " systemctl start 3CXPhoneSystemMC01 " and the management portal came up, but all my extensions are unregistered, and several services wont start.

 

[TLCtech]

When I try the command "sudo 3CXStartServices", this happens

pi@raspberrypi:~ $ sudo 3CXStartServices
Created symlink /etc/systemd/system/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Created symlink /etc/systemd/system/3CXCfgServ01.service → /lib/systemd/system/3CXCfgServ01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXCfgServ01.service → /lib/systemd/system/3CXCfgServ01.service.
Created symlink /etc/systemd/system/3CXMediaServer.service → /lib/systemd/system/3CXMediaServer.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXMediaServer.service → /lib/systemd/system/3CXMediaServer.service.
Created symlink /etc/systemd/system/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.
Created symlink /etc/systemd/system/3CXPhoneSystem01.service → /lib/systemd/system/3CXPhoneSystem01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystem01.service → /lib/systemd/system/3CXPhoneSystem01.service.
Created symlink /etc/systemd/system/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/3CXTunnel01.service → /lib/systemd/system/3CXTunnel01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXTunnel01.service → /lib/systemd/system/3CXTunnel01.service.
Created symlink /etc/systemd/system/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/3CXHotelModule01.service → /lib/systemd/system/3CXHotelModule01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXHotelModule01.service → /lib/systemd/system/3CXHotelModule01.service.
Job for 3CXHotelModule01.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status 3CXHotelModule01.service" and "journalctl -xe" for details.
Job for 3CXPhoneSystem01.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status 3CXPhoneSystem01.service" and "journalctl -xe" for details.
A dependency job for 3CXQueueManager01.service failed. See 'journalctl -xe' for details.
A dependency job for 3CXCallFlow01.service failed. See 'journalctl -xe' for details.
A dependency job for 3CXIVR01.service failed. See 'journalctl -xe' for details.
pi@raspberrypi:~ $

Journalctl shows this:

pi@raspberrypi:~ $ journalctl -xe
Nov 16 20:36:19 raspberrypi systemd[1]: Started Cleanup of Temporary Directories
-- Subject: A start job for unit systemd-tmpfiles-clean.service has finished suc
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit systemd-tmpfiles-clean.service has finished successfully
--
-- The job identifier is 2272.
Nov 16 20:38:55 raspberrypi PackageKit[4396]: daemon quit
Nov 16 20:38:55 raspberrypi systemd[1]: packagekit.service: Main process exited,
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit packagekit.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 15.
Nov 16 20:38:55 raspberrypi systemd[1]: packagekit.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit packagekit.service has successfully entered the 'dead' state.

More services are working now, but I still have a few that arent. Pic below. I am not sure what caused this.

 

[Andrew Whyte]

If nothings 'changed' before the reboot, I'd check the obvious - diskspace isn't full "df -k" make sure you're not at 100%. and make sure the time/date is accurate.

Beyond that, there are some commands mentioned in your output to run for more details.

>> See "systemctl status 3CXHotelModule01.service" and "journalctl -xe" for details.

you may need to be root to run journalctl -xe it is likely to spit a lot of stuff out.

 

[TLCtech]

My sd is about 50% full it looks like. Time and date are accurate

/dev/root 27896852 13659600 12797100 52% /
devtmpfs 3900372 0 3900372 0% /dev
tmpfs 4032468 42524 3989944 2% /dev/shm
tmpfs 4032468 18032 4014436 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 4032468 0 4032468 0% /sys/fs/cgroup
/dev/mmcblk0p6 258094 55170 202925 22% /boot
tmpfs 806492 0 806492 0% /run/user/1000
/dev/sda1 1953512444 7208556 1946303888 1% /media/pi/SAMBA

I ran it as root and this is what it spat out. Im really starting to freak out . We have no phones.

pi@raspberrypi:~ $ sudo -s
root@raspberrypi:/home/pi# journalctl -xe
-- Support: https://www.debian.org/support
--
-- A start job for unit 3CXPhoneSystem01.service has finished with a failure.
--
-- The job identifier is 2338 and the job result is failed.
Nov 16 20:43:26 raspberrypi systemd[1]: Dependency failed for 3CX PhoneSystem 01
-- Subject: A start job for unit 3CXCallFlow01.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit 3CXCallFlow01.service has finished with a failure.
--
-- The job identifier is 2277 and the job result is dependency.
Nov 16 20:43:26 raspberrypi systemd[1]: 3CXCallFlow01.service: Job 3CXCallFlow01
Nov 16 20:43:26 raspberrypi sudo[6436]: pam_unix(sudo:session): session closed f
Nov 16 20:44:50 raspberrypi sudo[6741]: pi : TTY=pts/1 ; PWD=/home/pi ; US
Nov 16 20:44:50 raspberrypi sudo[6741]: pam_unix(sudo:session): session opened f
Nov 16 20:44:50 raspberrypi sudo[6741]: pam_unix(sudo:session): session closed f
Nov 16 20:45:01 raspberrypi CRON[6791]: pam_unix(cron:session): session opened f
Nov 16 20:45:01 raspberrypi CRON[6792]: (root) CMD (command -v debian-sa1 > /dev
Nov 16 20:45:01 raspberrypi CRON[6791]: pam_unix(cron:session): session closed f
Nov 16 20:47:24 raspberrypi sudo[7282]: pi : TTY=pts/1 ; PWD=/home/pi ; US
Nov 16 20:47:24 raspberrypi sudo[7282]: pam_unix(sudo:session): session opened f

root@raspberrypi:/home/pi#

 

[Andrew Whyte]

The journalctl output isn't showing anything helpful. So not sure what the upstream cause is there.

In the 3CX Management Web UI - are you able to select the individual services and "start" them at all?
Assume they just fail again?

You may have some success looking at the logs in /var/lib/3cxpbx/Instance1/Data/Logs to determine what's failing and why.

 

[TLCtech]

At best, I get a pending on the services that arent starting


WHen I look in the Log folder, these are the files I get. Should I upload them here?

pi@raspberrypi:/var/lib/3cxpbx/Instance1/Data/Logs $ ls
3CXAudioProvider.2020-11-16.210328.log 3CXPhoneSystem.bldef
3CXCallFlow.log 3CXQueueManager.log
3CXConfService.2020-11-16.210318.log 3CXQueueManager.track.log
3CXIVR.log 3cxSystemService.log
3cxManagementConsole.log 3CXTunnel.2020-10-11.181242.log
3CXPhoneSystem-20201116-210242.blrec 3CXTunnel.2020-11-16.210312.log
3CXPhoneSystem-20201116-210328.blrec Backup

 

[TLCtech]

Could this have anything to do with my number port? Earlier in the day, I got notice to change my outbound and sip settings. So I did. Nothing happened for hours after. Could this be the issue?

 

[Andrew Whyte]

3cxSystemService.log is the most likely to show info around services.
You can tail the log file to watch it for new events. tail -150f 3cxSystemService.log and then try and start one of the failed services to see if it posts any useful messages. The SIP Server will be the one stopping phones logging in.

Changes to the SIP ports - perhaps yes. There are several of those network level settings that do NOT apply until services actually get restarted. It would be reasonable to undo those changes and restart/reboot to let it come back on previous settings.

Beyond that - I suspect the fastest way to get back to a working system would be to just reinstall and recover from a backup - assuming you've got a recent backup available to restore from.

 

[TLCtech]

I tried putting things back but it didnt help. And no, because I knew settings were about to change, I didnt bother doing a back up. I thought it was a simple number swap.

 

[TLCtech]

If I have to start over, I have to start over. Its been weeks of trying to get this right. BUt I appreciate all your help yall.

 

[Andrew Whyte]

Mmm bummer, sorry can't help more.

Even without a backup, you can probably save yourself some effort in re-building. While you still have access to the admin UI - You can at least Export the Extensions to allow re-importing them later. And Export Provider on the SIP Trunk.

I wouldn't suggest a full config backup at this point as it will no doubt restore the broken bits, but the Extensions, SIP Trunks, Contacts etc should all be individually exportable to save you some time putting them back.

 

[AWS2P]

as you didn't say, is your PI dedicated to PBX ?

 

[JohnS_3CX]

Hi,

We will need some more background info, any detail may make a difference here

- Pi model
- How was the installation done? exactly as per our guide using our linked image?
- Did you install anything else or modify and settings via commandline manually?
- Other than changing your DIDs, what other changes did you do recently, and I mean anything at all even updating via commandline

 

[TLCtech]

as you didn't say, is your PI dedicated to PBX ?

I may not understand this. Currently, its a local file server and a PBX system. We were trying to get it to serve as a vpn, but Im going to have to use another machine for that I think.

 

[TLCtech]

Hi,

We will need some more background info, any detail may make a difference here

- Pi model
- How was the installation done? exactly as per our guide using our linked image?
- Did you install anything else or modify and settings via commandline manually?
- Other than changing your DIDs, what other changes did you do recently, and I mean anything at all even updating via commandline

Pi 4 8gb
install was done with the instructions on the site, and using a video tutorial sponsored by 3cx, which gave me the idea. (

) Where this stopped, I picked back up using 3cx and then call centric tutorials. It was working yesterday when I walked into the office. I changed the DID and the sip trunk number because of the port. I have since changed these back and nothing has changed.

After changing that, I worked on installing openvpn on the pi. It doesnt look like its completely supported anyway. I gave up on that around 4:30. Phones and chat were working at that time. I rebooted the pi at this point. I didnt notice anything wrong for another couple of hours.

After getting back home and working more, I saw one fix was to simply update the programs. I ran apt-get upgrade and let it work. No dice.

 

[TLCtech]

Ill be backing up what I can (thanks Andrew. I wasnt sure if I could back anything up individually) and Ill keep an eye here. Thank you all again. This is my first go at this. I used to use 3cx at Sprint and knew it was the best out there. Something something educational, right?

as you didn't say, is your PI dedicated to PBX ?

Sounds like it SHOULD be dedicated, esp with a low power processor?

 

[TLCtech]

Im going to go ahead and reset this from scratch. Wish me luck

 

[TLCtech]

So I uninstalled and reinstalled everything. Im so glad its that easy. I got nearly everything back and what I fogot to save, we made new greetings for. AND because our number hasn't ported yet, our phone system is working. ROFL! Thanks for your help. I truly mean it.