Random System Service Interruptions

[Lee Cramman]

Over the last few days I've had the system service occasionally stop with the error message:

The following service(s) were interrupted: 3CXSystemService01

Along with people being unable to log in / out of hotdesking

There's nothing else shown in the event log. VPS is compliant with required specs (4Gb / 2vCPU / 2Gb swap), hosted by Google and has no third party stuff installed. Plenty of disk space (76Gb free). There has been a reboot in-between failures. Everything else on the dashboard shows green.

Currently running V20.0 Update 4 (Build 487 Release)

Syslog shows:

2025-03-10T12:15:20.891462+00:00 gc-custom-europe-west2-2021-10-03-16-49-45-5ltzh61al9t8w systemd[1]: 3CXSystemService01.service: Main process exited, code=dumped, status=6/ABRT
2025-03-10T12:15:21.043330+00:00 gc-custom-europe-west2-2021-10-03-16-49-45-5ltzh61al9t8w systemd[1]: 3CXSystemService01.service: Failed with result 'core-dump'.
2025-03-10T12:15:21.133195+00:00 gc-custom-europe-west2-2021-10-03-16-49-45-5ltzh61al9t8w systemd[1]: 3CXSystemService01.service: Consumed 1h 7min 3.808s CPU time.

In /var/lib/3cxpbx there is a core file time stamped for the same time.

If I manually start the service it comes up fine until the next time, which will probably be a few days.

I'm used to service issues usualy being down to running out of memory but free -h shows:

total used free shared buff/cache available
Mem: 3.8Gi 1.7Gi 1.1Gi 308Mi 1.6Gi 2.1Gi
Swap: 2.0Gi 504Mi 1.5Gi

Any ideas? I'm a bit stumped. Should I up the swap to 4Gb?

 

[fxbastler]

@Lee Cramman

If that is my understanding, then here are our experiences:
We know of these problems with a few of our on-premise 3CXs. We basically only run our 3CX VM with IPv4 and our own FQDN on premise. These problems were intangible, difficult or impossible to reproduce, and the cause could not be resolved in the 3CX VM itself. Upgrading the VM host has not always brought the desired success. That is our experience so far.

These 3CXs were all virtualized on small or medium-sized Proxmox or Hyper V hosts. The problems arose either because RAM sharing / ballooning was used or not deactivated, or because another VM on the same host briefly used too many system resources (CPU, Disk IO, short RAM increase - at least this was partially reproducible).

By allocating fixed / more resources or outsourcing the 3CX to another or even a separate host, these intangible problems have always been permanently resolved.

You can implement and test this yourself in the short term with few resources and it is worth a try.

 

[Lee Cramman]

I increased the swap after I posted (because, why not?), we'll see if that helps.

VPS is hosted by Google, not on-premise.

 

[fxbastler]

Increasing swap has never helped us in such a small 3CX environment. There must be enough real RAM (the amount shown for you is correct), the disk must be fast enough (IOPs and random read/write, hence only SSD), and there must be enough CPU idle time for real-time processing.
The latter was the case with our few problematic 3CX systems. There were times when, despite everything else, there weren't enough CPU resources available for very short periods. Even if the host was only at 10% load 90% of the time, the 3CX crashed during reproducible peak times. We noticed this when we integrated the system (host, 3CX VM and some other VMs) into our Zabbix and increased the monitoring rate.

VPS is hosted by Google, not on-premise.

I know, your wrote it. This somewhat limits the possibilities for monitoring.

 

[Lee Cramman]

Increasing swap has never helped us in such a small 3CX environment. There must be enough real RAM (the amount shown for you is correct), the disk must be fast enough (IOPs and random read/write, hence only SSD), and there must be enough CPU idle time for real-time processing.
The latter was the case with our few problematic 3CX systems. There were times when, despite everything else, there weren't enough CPU resources available for very short periods. Even if the host was only at 10% load 90% of the time, the 3CX crashed during reproducible peak times. We noticed this when we integrated the system (host, 3CX VM and some other VMs) into our Zabbix and increased the monitoring rate.


I know, your wrote it. This somewhat limits the possibilities for monitoring.

I'll continue to monitor, but no further issues so far.