CPU Over utilization led to broken instance

[drewt]

Lightsail hosted Standard Annual 16.0.5.619 using SBC's (x2) Customer has 6 yealink T46 handsets all on the lateset firmware, and 1 cell phone app and 1 windows client. Instance manager and dashboard show 100% utilization. Top 3 cpu process's are as follows:


/usr/lib/3cxpbx/createdump --name core.minidump -- 93.7%,

/bin/bash /var/lib/3cxpbx/InstanceManager/check_al 1.2%,

/usr/lib/3cxpbx/3CXManagementConsole --pid /var/ru 0.4%,

Customer has reported recent problem with SP button blf status not behaving correctly. Waiting for feedback from customer. Anyone have any insight? I have 13 other instances configured in the same manner, and they are having no issues. This customer has 2 raspi SBC's that have not been updated, same as all other 13 instances. Undstand that SBC update is still in beta.

Thanks,
Drew

 

[cobaltit]

Seems like a reboot would be in order to clear that dump process.

 

[drewt]

I would like to reboot the lightsail instance, whats best way to reboot from lightsail console, I was thinking stop the 3cx services, then reboot the lightsail debian instance.

 

[cobaltit]

You can just go to the 3CX management console and use the terminal to reboot.

 

[drewt]

Does that reboot the debian vm instance or just the 3cx process? I will try rebooting from the instance manager for the practice. If that doesnt bring cpu utililization back into line I will reboot from lightsail ssh. Thank you for you input!

 

[drewt]

Using instance manager to reboot was not successful. Rebooted using dashboard terminal. After terminal reboot tried to log into web gui, got "server error" message in web gui auth screen. Rebooted instance in lightsail control panel, same result, got "server error" message in web gui auth screen. Only other choice was to stop the instance in the lightsail control panel. Started the instance backup and got the same result, "server error" message in web gui auth screen. Instance manager shows cpu utilization dropping back into green but still no response from web gui, purchased support tic from 3cx.

Top 3 CPU/bin/bash /var/lib/3cxpbx/InstanceManager/check_al 3.0%, /usr/lib/3cxpbx/3CXManagementConsole --pid /var/ru 1.5%, /usr/lib/3cxpbx/3cxSystemService -p /var/run/phone 1.0%,Top 3 Directories148M /var/lib/3cxpbx/Instance1/Data, 148M /var/lib/3cxpbx/Instance1, 102M /var/lib/3cxpbx/Instance1/Data/Http,Top 3 Memory/usr/lib/3cxpbx/3CXManagementConsole --pid /var/ru 13.6%, /usr/lib/3cxpbx/3cxSystemService -p /var/run/phone 11.3%, /usr/lib/3cxpbx/3CXCallFlow -p /var/run/phonesyste 10.6%,Top 3 Swapnone

 

[drewt]

Reboot at the lightsail control panel brought it back up so I could connect with SSH, but never got webgui back. Purchased trouble ticket from 3cx, no response 11 hours later, ran pbx express, dns came right back up. Logged in, connected the backup to the ftp location, uploaded the backup from the now deprecated google drive. One note, when I uploaded the .zip backup file to the new ftp location, the ftp file manager appended my .zip backup file name with .temp 3cx backup manager could not see the backup file until I renamed the file and removed the .temp.