Heads up: Lightsail-based instances break on updating kernel

[GordonShumway]

I've sent a message to your support, but this is just a heads up for a larger audience.

If you update the kernel on the Lightsail (maybe AWS instances are susceptible too) v16 instance it will show up as running but no SSH/HTTPS/SIP connections will be possible.

Steps to reproduce:
1. Use Express setup to roll a brand new 3cx instance on AWS.
2. Run

sudo apt-get update; sudo apt-get upgrade; sudo apt-get install linux-image-amd64; sudo reboot;

3. Ensure that neither SSH nor HTTPS, SIP connection is possible.

I have (sadly) tried it with both alpha and current release versions of 3cx.

PS. This is happening in us-west2 region.

 

[Evolute IT]

And this is why 3CX recommends NEVER running "apt upgrade" ourselves.

Let the 3CX manage it.

 

[Mic_Mac]

we just lost the fleet of 50 LightSail instances we host. All updated to Debian 4.9.0.14 overnight and we lost HTTPs and SSH access - 4 hours with AWS support still left us unrecoverable instances. Today we are building instances and restoring from backups. been a long day, more to come unfortunately.
MAKE SURE OS Upgrades are not automated! we are sure ours was not set to auto update, but somehow they did.
please check your updates are good. as this may bite many more ahead.
We are in AU Syd region

 

[hogan71088]

So just to confirm, this was a manual update to Debian and nothing to do with 3CX auto updates from the MGT console...

 

[cobaltit]

we just lost the fleet of 50 LightSail instances we host. All updated to Debian 4.9.0.14 overnight and we lost HTTPs and SSH access - 4 hours with AWS support still left us unrecoverable instances. Today we are building instances and restoring from backups. been a long day, more to come unfortunately.
MAKE SURE OS Upgrades are not automated! we are sure ours was not set to auto update, but somehow they did.
please check your updates are good. as this may bite many more ahead.
We are in AU Syd region

Ouch. Sorry to hear about this. At least you have backups to recovery from. Hopefully you get everything back up quickly!

 

[LeonMuller]

we just lost the fleet of 50 LightSail instances we host. All updated to Debian 4.9.0.14 overnight and we lost HTTPs and SSH access - 4 hours with AWS support still left us unrecoverable instances. Today we are building instances and restoring from backups. been a long day, more to come unfortunately.
MAKE SURE OS Upgrades are not automated! we are sure ours was not set to auto update, but somehow they did.
please check your updates are good. as this may bite many more ahead.
We are in AU Syd region

We also experienced a similar issue and we were curious if others have.

After this issue happened we were able to convert our machines to boot on a non HVM Kernel to regain access and check the status of everything. What we found was:

1) The daily apt update and upgrade crons were disabled
2) unattended upgrade was set to "no"
3) unattended upgrade package was in fact what upgraded the kernel security update

Digging further the logs did not show what triggered this and a dependency check showed the unattended upgrade package was installed and bound to the 3cxphonesystem.

Is this how 3CX pushes patches and security updates and was this what triggered the kernel security update?

Luckily changing the boot kernal and debian fixing the back-port issue with the update allowed us to correct the kernal panic, updating to the "in response" patch from debian.

Glad we were not using amazon as it sounds like they dont allow you to have this control

 

[GordonShumway]

Glad we were not using amazon as it sounds like they dont allow you to have this control

It's been a while since I've had to do that and I may be confusing Google Compute Cloud and AWS, but I believe there is a serial access available for AWS, but not Lightsail.

Either way, if 3cx depends on specific packages/kernels that should not be updated, it should be managed/marked as such thru the package management rather than relying on users/automated processes not to run updates.

 

[N_G]

Yes, you should not have run those commands upon installation. Our instructions are very clear. If you upgrade the kernel of an operating system you cant expect the software running on top of it to keep running flawlessly. So either follow instructions and ensure you do not upgrade or set auto upgrade, or let us manage operating system updates and choose hosted by 3CX.

 

[aweedman]

My Lightsail instance has been down since this past Friday. I just noticed it today. Can't SSH in or ping the server. I did not do a manual OS update. Any suggestions on steps to restore from Thursday's backup?

 

[aweedman]

I can copy my Lightsail sever to EC2. From there it looks like I can follow this procedure to revert the kernel and retrieve the backup file:

https://aws.amazon.com/premiumsupport/knowledge-center/revert-stable-kernel-ec2-reboot/

What is the kernel version that I should use?

 

[JohnS_3CX]

My Lightsail instance has been down since this past Friday. I just noticed it today. Can't SSH in or ping the server. I did not do a manual OS update. Any suggestions on steps to restore from Thursday's backup?

You can redeploy straight from your customer portal, if you choose self hosting


You can also take a look at our guide here: https://www.3cx.com/docs/cloud-pbx-amazon-aws/

 

[GordonShumway]

My Lightsail instance has been down since this past Friday. I just noticed it today. Can't SSH in or ping the server. I did not do a manual OS update. Any suggestions on steps to restore from Thursday's backup?

If you have backups outside of that instance, I'd redeploy and restore from backup. If not, export the snapshot to AWS, run another minimal instance, attach the drive from the snapshot to said instance, recover your backups and redeploy/restore from backup.

 

[GordonShumway]

Yes, you should not have run those commands upon installation. Our instructions are very clear.

May I suggest you customize the banner/motd and include this important bit of information there? I've been using self-hosted 3cx for years, I've had no idea 3cx manages other packages/OS updates.

If you upgrade the kernel of an operating system you cant expect the software running on top of it to keep running flawlessly.

Just wanted to preserve this epic quote here for posterity.