100% CPU on Azure - GRUB needs to reinstall

[SKDamon]

My 3CX Install has auto-updates enabled, when I came into office this morning we had issues with call audio.
After a brief look at the Management Console I found the CPU was utilized 100%, caused by python running unattended-upgrades.

Manually running apt-get dist-upgrade resolved the CPU load issue, but I was prompted to specify the boot device / grub install location
because "The GRUB boot loader was previously installed to a disk that is no longer present"...

I chose the first disk which contains my root partition (not the partition itself), can anybody confirm this is the device Azure expects to boot from?
The VM was deployed using the 3CX PBX Express tool. The VM previously ran for months without problems.

3CX Version: 16.0.910
Debian GNU/Linux 9

From the upgrade log in terminal:

Preparing to unpack .../0-grub-pc_2.02~beta3-5+deb9u2_amd64.deb ...
Unpacking grub-pc (2.02~beta3-5+deb9u2) over (2.02~beta3-5+deb9u1) ...
Preparing to unpack .../1-grub-pc-bin_2.02~beta3-5+deb9u2_amd64.deb ...
Unpacking grub-pc-bin (2.02~beta3-5+deb9u2) over (2.02~beta3-5+deb9u1) ...
Preparing to unpack .../2-grub2-common_2.02~beta3-5+deb9u2_amd64.deb ...
Unpacking grub2-common (2.02~beta3-5+deb9u2) over (2.02~beta3-5+deb9u1) ...
Preparing to unpack .../3-grub-common_2.02~beta3-5+deb9u2_amd64.deb ...
Unpacking grub-common (2.02~beta3-5+deb9u2) over (2.02~beta3-5+deb9u1) ...

 

[SKDamon]

Today the unattended-upgrades package consumed 100% CPU again while installing the linux kernel over and over...

Log started: 2019-09-25 11:45:20
(Reading database ... 40754 files and directories currently installed.)
Preparing to unpack .../linux-image-4.9.0-11-amd64_4.9.189-3+deb9u1_amd64.deb ...
Unpacking linux-image-4.9.0-11-amd64 (4.9.189-3+deb9u1) over (4.9.189-3+deb9u1) ...
Setting up linux-image-4.9.0-11-amd64 (4.9.189-3+deb9u1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.9.0-11-amd64
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.9.0-11-amd64
Found initrd image: /boot/initrd.img-4.9.0-11-amd64
Found linux image: /boot/vmlinuz-4.9.0-9-amd64
Found initrd image: /boot/initrd.img-4.9.0-9-amd64
done
Log ended: 2019-09-25 11:46:07

Log started: 2019-09-25 11:46:12
(Reading database ... 40754 files and directories currently installed.)
Preparing to unpack .../linux-image-4.9.0-11-amd64_4.9.189-3+deb9u1_amd64.deb ...
Unpacking linux-image-4.9.0-11-amd64 (4.9.189-3+deb9u1) over (4.9.189-3+deb9u1) ...
Setting up linux-image-4.9.0-11-amd64 (4.9.189-3+deb9u1) ...

Status after killing unattended-upgrades:

~# dpkg -l linux-image*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii linux-image-4.9.0-11-amd64 4.9.189-3+deb9u1 amd64 Linux 4.9 for 64-bit PCs
rc linux-image-4.9.0-7-amd64 4.9.110-3+deb9u2 amd64 Linux 4.9 for 64-bit PCs
rc linux-image-4.9.0-8-amd64 4.9.144-3.1 amd64 Linux 4.9 for 64-bit PCs
ii linux-image-4.9.0-9-amd64 4.9.168-1+deb9u5 amd64 Linux 4.9 for 64-bit PCs
ii linux-image-amd64 4.9+80+deb9u9 amd64 Linux for 64-bit PCs (meta-package)

linux-image-4.9.0-11 is installed and apt-get install -f does not install/configure any packages...

 

[JohnS_3CX]

Hi SKDamon,

Is this a very small machine on a single core?

 

[SKDamon]

Yes the machine size is Standard_A1_v2 (1 vCPU, 2GB RAM)
The problem persists after multiple manual system updates (apt-get dist-upgrade) and reboots.

Did Hardware requirements change since V16 release? The VM ran fine w/ V15 and V16 up until a few weeks ago...

EDIT: The latest 3CX Update was seamlessly installed last night, but unattended-upgrades still loops

2019-10-04 12:01:39,265 DEBUG found partition of size 13 (['libnginx-mod-http-geoip', 'libnginx-mod-http-upstream-fair', 'libnginx-mod-http-auth-pam', 'libnginx-mod-http-dav-ext', 'nginx-common', 'libnginx-mod-http-subs-filter', 'nginx-full', 'libnginx-mod-http-image-filter', 'libnginx-mod-http-echo', 'libnginx-mod-mail', 'libnginx-mod-stream', 'libnginx-mod-http-xslt-filter', 'nginx'])
2019-10-04 12:01:41,243 DEBUG applying set ['libnginx-mod-http-geoip', 'libnginx-mod-http-upstream-fair', 'libnginx-mod-http-auth-pam', 'libnginx-mod-http-dav-ext', 'nginx-common', 'libnginx-mod-http-subs-filter', 'nginx-full', 'libnginx-mod-http-image-filter', 'libnginx-mod-http-echo', 'libnginx-mod-mail', 'libnginx-mod-stream', 'libnginx-mod-http-xslt-filter', 'nginx']
2019-10-04 12:01:53,838 DEBUG left to upgrade {'systemd'}
2019-10-04 12:01:55,590 DEBUG found partition of size 13 (['libnginx-mod-http-geoip', 'libnginx-mod-http-upstream-fair', 'libnginx-mod-http-auth-pam', 'libnginx-mod-http-dav-ext', 'nginx-common', 'libnginx-mod-http-subs-filter', 'nginx-full', 'libnginx-mod-http-image-filter', 'libnginx-mod-http-echo', 'libnginx-mod-mail', 'libnginx-mod-stream', 'libnginx-mod-http-xslt-filter', 'nginx'])
2019-10-04 12:01:57,544 DEBUG applying set ['libnginx-mod-http-geoip', 'libnginx-mod-http-upstream-fair', 'libnginx-mod-http-auth-pam', 'libnginx-mod-http-dav-ext', 'nginx-common', 'libnginx-mod-http-subs-filter', 'nginx-full', 'libnginx-mod-http-image-filter', 'libnginx-mod-http-echo', 'libnginx-mod-mail', 'libnginx-mod-stream', 'libnginx-mod-http-xslt-filter', 'nginx']
2019-10-04 12:02:09,941 DEBUG left to upgrade {'systemd'}

 

[SKDamon]

I believe this issue is fixed in a newer version of unattended-upgrades:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=899366

Unfortunately, Debian Stretch uses an older version and only gets security backports.
And 3CX does not ship packages specifically for Debian Buster yet.
I'll disable auto-updates until Debian Buster is on the supported OS list.

 

[JohnS_3CX]

Hi @SKDamon

The PBX Express instances from the past may not have the swapfile enabled. We changed this a few months ago to include it on any future deployments, but yours may be before that time. You can run the following commands to enable it. This can potentially help mitigate the issue in case it is looping due to lack of memory:

sudo apt update

sudo apt install net-tools dphys-swapfile

ignore those above commands because you are on Azure

https://support.microsoft.com/en-us...d-a-swap-file-in-linux-azure-virtual-machines

 

[JohnS_3CX]

You can also ensure that everything is setup correctly if you use a machine from the Azure Marketplace (which we ensure gets configured correctly)

Take a backup off of your existing machine, then scrap that machine and start anew using your backup with the below:

https://azuremarketplace.microsoft.com/en-us/marketplace/apps/3cx-pbx.3cx-pbx?tab=Overview