3cx update started on schedule and took the system down

rayno_b

Gold Partner
Joined
Apr 6, 2017
Messages
9
Reaction score
1
Our 3cx system is set to automatically run updates.

Our 3CX update (20.0.10.1785) also triggered a Debian OS update that included grub-pc. Because grub-pc.postinst spawned an interactive debconf frontend prompt asking which disk (/dev/nvme0n1) to install GRUB onto—and there was no interactive terminal attached to the background 3CX updater—dpkg and apt-get have been sitting frozen on that prompt, holding the package lock while 3cxpbx is half-upgraded.

This was quite frustrating, as the dependency on 3cx to manage these auto-updates, effectively failed half way, leaving the entire system inaccessible until we manually found the issue, got the grup-pc matter resolved and finalising the update.

My question is - is this a common issue, and why did the update trigger if it needed an interactive prompt on an administrator side to finalise the update? This caused quite a considerable amount of down time for the end users and was extremely difficult to figure out.
 
Solution
I can confirm the problem. this issue happened on 2 of my 3CX installations.
system was stuck with 60% cpu

# ps aux | grep -E "apt|dpkg"
root 135153 0.0 0.1 6340 2080 pts/1 S+ 08:18 0:00 grep -E apt|dpkg
root 1179160 0.3 0.5 19372 12028 ? S 05:24 0:34 apt-get -y --allow-downgrades install 3cxpbx=20.0.10.1785
root 1179274 0.0 0.3 10536 6060 pts/0 Ss+ 05:24 0:00 /usr/bin/dpkg --status-fd 17 --configure --pending
root 1179275 8.0 1.0 25512 20316 pts/0 R+ 05:24 14:04 /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/grub-pc.postinst configure 2.06-13+deb12u1
root 1179308 7.6 0.1 7336 3716 pts/0 S+ 05:24 13:24 /bin/bash...
You'll need to fix the grub installer issue or you'll encounter this again.

Code:
debconf-show grub-pc | grep install_devices

check this is your disk

# debconf-show grub-pc | grep install_devices
grub-pc/install_devices_empty: false
* grub-pc/install_devices_disks_changed: /dev/vda
* grub-pc/install_devices: /dev/vda
grub-pc/install_devices_failed: false
grub-pc/install_devices_failed_upgrade: true

# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
udev 984712 0 984712 0% /dev
tmpfs 201416 540 200876 1% /run
/dev/vda1 29801344 8578136 19684036 31% /
tmpfs 1007060 1056 1006004 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 201412 0 201412 0% /run/user/0

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sr0 11:0 1 1024M 1 rom
vda 254:0 0 30G 0 disk
├─vda1 254:1 0 29G 0 part /
├─vda2 254:2 0 1K 0 part
└─vda5 254:5 0 975M 0 part [SWAP]

In the interactive grub install I selected VDA as the grub target.

can you please explain what you mean with -> You'll need to fix the grub installer issue or you'll encounter this again.
 
Our 3CX server suffered a similar fate. I've rolled it back to before the update and disabled Auto updates. Is this something that 3CX will release an update to fix or will we need to fix it ourselves in order to get back on track. We don't have much linux knowledge in house (enough to be dangerous). A step by step guide would be helpful if anyone is feeling very generous.