3CX v15.5.6354.2 Debian 9 CPU 100%

[ucme4now]

I have a client with a brand new install (120+) users running on VMware v6.5, everything has been going well until recently the CPU in the VM spikes to 100% utilization and stays there. Of course this knocks all of the users off the system. This has happened twice this week around the same time frame each both days.

I see this CRON job kick off several times prior to the CPU spike:

CRON[31799]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Any ideas what that CRON is doing?

When the CPU locks up this is what the log file shows that sticks out:

"NULNULNULNULNULNULNULNULNULNULNUL" this repeats about (50) more times...

I don't see anything obvious in the log files. Has anyone else see anything like this? Or is this more of a Debian OS issue? Generally the CPU hovers around 3-5% utilization and there is 4G of RAM with roughly 20% utilization there.

I have other clients running the same basic config with no issues. Any and all help and feedback much appreciated.

 

[ucme4now]

Did you install from the 3CX ISO?

Yes both times. Had no choice, it didn't look like the VM was going to recover on its own.

 

[ucme4now]

Did you install from the 3CX ISO?

I did. I have found that is the easiest way.

 

[ucme4now]

The sa1 command collects and stores binary data at intervals. The cron task is defined in /etc/cron.d/sysstat
I've not seen this behaviour before with this task running. I assume you've tried a reboot
The command man sa1 will help you here

The command: "man sa1" comes back with "command not found"

 

[jimbo59]

If you get nothing with "man sa1," I'd suggest backup the system to google cloud and reinstall.

 

[ucme4now]

Interesting, I checked a couple of other recent ISO installs and the "man sa1" doesn't work there either.

 

[jimbo59]

Mine does. I have a standard ISO from 3CX and I've not changed a thing.
Won't hurt to do an "apt-get install man-db" though before reinstall.

 

[ucme4now]

So, I just pulled the 3CX Debian ISO file down and reinstalled. No MAN pages. Am I missing something???

 

[jimbo59]

Strange.
just do apt-get install man-db and move on.

 

[cobaltit]

So, I just pulled the 3CX Debian ISO file down and reinstalled. No MAN pages. Am I missing something???

I assume this is part of 3CX slimming things down to be more micro cloud instance friendly.

 

[ucme4now]

With any luck, this is the root cause of the 100% CPU spike. The is an issue with VMware and Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864642

https://www.reddit.com/r/homelab/comments/6s5dh6/debian_9_on_esxi_65u1_complete_lockup/

After (4) crashes this week with the fourth happening to my test unit with zero load on it, something had to give... The VMware 10G virtual driver has an issue with certain Debian releases, read the articles. Changing the virtual adapter back to the 1G e1000 seems to be the work around.

Changes have been made and will be monitored. Hopefully everything runs clean next week because I'll be in Ft Lauderdale at the ITExpo Show!