3CX PhoneSystem 01 System Server

[spuky]

Seit heute zickt unsere 3cx

der Systemserver beendet sich:

in der mail steht:
The following service(s) were interrupted:
3CXSystemService01

Installation:

Aws Lightsail
- 1gb (laut dashbord 293.1 MB frei)
- 40gb disk 30.5 GB frei

sudo service 3CXSystemService01 start

Job for 3CXSystemService01.service failed because a timeout was exceeded.
See "systemctl status 3CXSystemService01.service" and "journalctl -xe" for details.

journalctl -xe:

-- A start job for unit 3CXSystemService01.service has begun execution.--
-- The job identifier is 691.
Aug 14 13:40:56 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Start operation timed out. Terminating.
Aug 14 13:40:56 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Main process exited, code=killed, status=2/INT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit 3CXSystemService01.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 2.
Aug 14 13:40:56 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit 3CXSystemService01.service has entered the 'failed' state with result 'timeout'.
Aug 14 13:40:56 ip-172-26-0-138 systemd[1]: Failed to start 3CX PhoneSystem 01 System Server.
-- Subject: A start job for unit 3CXSystemService01.service has failed-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit 3CXSystemService01.service has finished with a failure.
--
-- The job identifier is 691 and the job result is failed.

Nach einen reboot wird der Dienst kurz als laufend angezeigt... stirbt dann aber...
Hat jemand ne Idee was sein kann ?

 

[spuky]

I don't know how that got into the englisch form...
since today:


The following service(s) were interrupted:
3CXSystemService01

tried to start on the comandline but it somehow times out..

after a reboot the service is shown as running seems to die later on...

 

[SweetAction]

Officially you need 2GB RAM. I would check syslog to see why the service won't start.
Do you have a swap file?

 

[spuky]

syslog gives not much more infor...:

Aug 14 15:48:12 ip-172-26-0-138 systemd[1]: Starting 3CX PhoneSystem 01 System Server...
Aug 14 15:49:28 ip-172-26-0-138 dhclient[413]: XMT: Solicit on eth0, interval 116650ms.
Aug 14 15:49:42 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Start operation timed out. Terminating.
Aug 14 15:49:42 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Main process exited, code=killed, status=2/INT
Aug 14 15:49:42 ip-172-26-0-138 systemd[1]: 3CXSystemService01.service: Failed with result 'timeout'.
Aug 14 15:49:42 ip-172-26-0-138 systemd[1]: Failed to start 3CX PhoneSystem 01 System Server.

Swap:

[CODE]cat /proc/swaps
Filename    Type    Size    Used    Priority
/var/swap   file    2035708    322816    -2

[/CODE]

the instance was installed with the asistant before there was the cloud offering by 3cx in 2020

 

[SweetAction]

journalctl in verbose show anything helpful?

journalctl -u 3CXSystemService01.service -r -o verbose

 

[spuky]

nothing helpful besides the timeout information...

did some more digging since it is a timeout i suspect some network stuff...

during the time the service tries to restart I see 2 unanswerd outgoing connections to cloudflare...
so i guess some 3cx Service... that does not answer...

 

[Lee Cramman]

Problems with services are most frequently caused by insufficient RAM as per @SweetAction . You have less than the recommended amount of RAM.

A swap file might help you get the service started but you should really allocate more RAM.

 

[spuky]

The instance Type and Ram was chosen by 3cx for there we install a lightsail instance for you !
So don't fault me for that ;-)

 

[Lee Cramman]

The instance Type and Ram was chosen by 3cx for there we install a lightsail instance for you !
So don't fault me for that ;-)

Not trying to fault you , but it is a fact:

https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/

Why not stop your instance, add some more RAM, start it again and see if the problem goes away?

I have seen installs get away with less than 2Gb (although it's not supported), but they've needed to use a swap file.

 

[spuky]

Ok Service shows the same behavior on a 2GB 60G instance...

 

[SweetAction]

If this is a VM / you have a spare VM and you have a full backup, can you do a quick snapshot, reinstall, and restore? Takes about 15 minutes and would help rule out networking being the cause - if the new VM works, great, and if not you ruled out the configuration of the VM as the issue and can always roll back to the snapshot/other VM.

 

[spuky]

tried a restore a backup from august 7 before 18.0 update 8 (it seems the problem came with the update... since a mail about a successfull update there where no backups done...) the on a new install same behavior "3CX PhoneSystem 01 System Server" fails a few minutes after boot..

downgraded to 18.0.7.441 and the service is working...

 

[SweetAction]

Based on what you wrote, it seems like the service can start on a new install but dies after a few minutes, just like the existing install. This issue only happens with the new versions of 3CX and older versions work just fine.

I think without better logs (since the service is running you can also check 3CX specific logs for last thing before service stops) you will be stuck asking 3CX for additional help. It doesn't seem to be lightsail specific.

 

[spuky]

18.0.7.441 the service is up within seconds... on 18.0.8.912 and 18.0.8.917 the service would be reported as up as long as it's startuptime (gave it 900 secs and rebooted the machine) .. so even during boot it is not finishing the service startup.

if there where good docs on which logfile is for that service... I'd be happy to check what is happening.

For now I will stay on 18.0.7.441 an see if there are more fixes coming to 18.0.8

 

[SweetAction]

Look at the system log here: var / lib / 3cxpbx / Instance1 / Data / Logs / 3cxSystemService.log