Toggle sidebar Toggle sidebar

Periodic 100% cpu usage

Status
Not open for further replies.

steve_HR

Free User
Joined
Feb 11, 2019
Messages
6
Reaction score
0
Greetings,
I have Version 16.0.1.273 3CX hosted trial account. Using Twilio as SIP provider. 2SBC and 7 phones connected
I see 100% CPU usage every hour @30 minutes past the hour. A few days ago it seemed to be happening every 6 hours.
This results in the audio stuttering until after a minute or two. Then when the CPU usage goes back to the usual 4% the audio stops stuttering. I have purged the call history and chat logs, I have not purged the event log.
100% CPU usage occurs even if no calls are taking place.

I'm not sure where to look in the logs or test.
 
I've same behavior on two cloud hosted one on OVH other on Lightsail, when nothing happens ,no calls, no phones connected, really nothing done on phone system and suddenly the CPU goes to 80% or 100% and few seconds after it goes back to 0
 
But is it a 3CX Process taking 100% CPU usage?
 
I had this issue on one of our servers, and it was the apt-cache process that was actually ramping up - and all weekend had been peaking and going back down, restarts etc did nothing.

I ran the following: -

sudo apt-get update
sudo apt-get upgrade

And so far it has been okay, obviously only worth trying this if you are confident its an issue similar to mine with apt causing the 100% CPU usage.
 
Dear all, same issue here as reported by @rocketuc

My 3CX installation is on a Debian 9 machine.

top output

top - 10:22:28 up 1:08, 1 user, load average: 6.03, 5.85, 4.98
Tasks: 168 total, 7 running, 161 sleeping, 0 stopped, 0 zombie
%Cpu(s): 98.4 us, 1.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 2047904 total, 209688 free, 818956 used, 1019260 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 1034508 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6545 instanc+ 20 0 78296 42432 5596 R 16.1 2.1 22:15.71 apt-cache
7922 instanc+ 20 0 73544 32712 5644 R 16.1 1.6 12:36.82 apt-cache
10603 instanc+ 20 0 70476 26612 5616 R 16.1 1.3 7:15.56 apt-cache
12631 instanc+ 20 0 67196 19972 5564 R 16.1 1.0 2:59.91 apt-cache
14316 instanc+ 20 0 65152 15468 5548 R 16.1 0.8 1:07.93 apt-cache
9459 instanc+ 20 0 70872 27264 5476 R 15.8 1.3 7:53.80 apt-cache
730 phonesy+ 20 0 3162256 161752 75480 S 1.3 7.9 1:09.08 3CXManagementCo
454 phonesy+ 20 0 687860 11028 9152 S 0.6 0.5 0:14.74 3CXTunnel
14632 root 20 0 44908 3732 3116 R 0.6 0.2 0:00.03 top
229 root 20 0 134052 10764 9420 S 0.3 0.5 0:02.43 vmtoolsd
453 phonesy+ 20 0 2929592 65348 45524 S 0.3 3.2 0:19.41 CloudServicesWa
651 postgres 20 0 142816 3868 1940 S 0.3 0.2 0:00.14 postgres
729 phonesy+ 20 0 755400 22652 13052 S 0.3 1.1 0:17.35 3CXPhoneSystem
731 phonesy+ 20 0 345008 5184 4608 S 0.3 0.3 0:06.42 3CXAudioProvide
768 phonesy+ 20 0 884096 177032 13480 S 0.3 8.6 0:20.21 3CXIVR
11376 root 20 0 99348 6868 5880 S 0.3 0.3 0:00.05 sshd
1 root 20 0 57288 6944 5268 S 0.0 0.3 0:00.98 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.34 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kworker/u64:0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.69 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
19 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
21 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
23 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
24 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq
27 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 watchdogd
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 vmstat
41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
42 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ipv6_addrconf
75 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
76 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
77 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
78 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset


ps -aux | grep apt-cache output

root@VPS-3CX:~# ps -aux | grep apt-cache
instanc+ 6545 38.2 2.0 78296 42696 ? R 09:25 22:28 apt-cache policy 3cxpbx
instanc+ 7922 26.3 1.6 73676 32976 ? R 09:35 12:49 apt-cache policy 3cxpbx
instanc+ 9459 20.9 1.3 70872 27528 ? R 09:45 8:06 apt-cache policy 3cxpbx
instanc+ 10603 20.4 1.3 70608 26876 ? R 09:47 7:28 apt-cache policy 3cxpbx
instanc+ 12631 17.3 0.9 67460 20236 ? R 10:05 3:12 apt-cache policy 3cxpbx
instanc+ 14316 15.9 0.7 65416 15996 ? R 10:15 1:20 apt-cache policy 3cxpbx
root 14686 0.0 0.0 12780 936 pts/0 S+ 10:23 0:00 grep apt-cache

It was a couple of days I'm facing this issue.

Best regards
 
Unfortunately, my fix only lasted a few hours - back up to 100% now, with apt-cache back at 96%.
 
File on which apt-cache policy command is called is /var/lib/3cxpbx/InstanceManager/check_all.sh

I really do not know why it stucks, because if I run
# apt-cache policy 3cxpbx

it returns me correct output in few seconds
root@VPS-3CX:~# apt-cache policy 3cxpbx | grep "Candidate: " | head -1 | tr -s ' ' | cut -d" " -f3
16.0.1.273
 
The Services page in the management console during the 100% cpu usage events shows no more than 14% CPU usage. 14% was in SIP Server and SIP/RTP Tunneling Proxy services.
 
Thank you Steve_HR
as you can see Services have very low CPU usage.
While on dashboard you can see CPU is 100%
 

Attachments

  • cpu.png
    cpu.png
    57.9 KB · Views: 37
  • services.png
    services.png
    42.2 KB · Views: 37
MichaelB said:
But is it a 3CX Process taking 100% CPU usage?
Click to expand...
While looking at the Services tab during a 100% CPU usage event I saw "3CX PhoneSystem Database Server" go to 93 CPU Usage for a moment. SIP Server was 14 CPU usage. All this when the system was no processing any calls. I am now running version 16.0.2.900
 
When it happens, make sure:
  • you have enough free disk space
  • if your disk is network attached make sure it doesn't lose connectivity
  • you have enough free memory (create swap)
Might seem irrelevant but in some cases they cause such problems.
 
Just to aware you about issue I faced: my issue was related to Instance Remote Management.
Deactivating it CPU stays at 1-2% forever.
Best regards
 
steve_HR said:
While looking at the Services tab during a 100% CPU usage event I saw "3CX PhoneSystem Database Server" go to 93 CPU Usage for a moment. SIP Server was 14 CPU usage. All this when the system was no processing any calls. I am now running version 16.0.2.900
Click to expand...

Hi @steve_HR

This could be answered in a matter of seconds. In Linux you can run the top command, in Windows you open the task manager. Find out what process is eating up the CPU and we can move forward from there.
 
We're also experiencing the same issue. Top shows apt-cache using 99.9% CPU
 
RussellPTown said:
We're also experiencing the same issue. Top shows apt-cache using 99.9% CPU
Click to expand...

To clarify, we're running the 3CX v16 Debian install in house. I can kill the process(es) to return CPU usage to normal, but the process will respawn some time later and again consume all available CPU.
 
I had the same problem with about 40 of our hosted 3cx v16 Debian PBXs. Disabling the instance manager on the PBX fixed this problem. After killing all apt-cache and then salt-minion processes I could safely disable the instance manager.
 
Fixed! The cause seemed to be the 3CX PhoneSystem Database service's periodic activity and a lack of free memory on the 3CX hosted trial. I moved the system to a Google Cloud VM, n1-standard-1, with 3.75 GB memory. No more 100%CPU usage events!

It was difficult to find because I did not have shell access to the trial instance. Being able to run top would have helped.

Thank you to all for the help in finding the cause.
 
Hi @steve_HR

I'm glad it was resolved. Please note that the free trial hosting is running on smaller machines so it is normal to have low resources as they are meant for evaluation rather than production.
 
Hi
Yes I am, i've now removed instance manager
 
Last edited:
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
112,089
Messages
590,697
Members
165,054
Latest member
jmc-tim
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom