Linux Upgrade Aborted - Troubleshooting tips

[B.lueken]

Overall, the 3CX upgrade script upgrades Linux based customers from Debian 9 with v16 to Debian 10 with v18 with a success rate of 92%. In this blog post, we will cover common solutions for the remaining 8% of cases that failed to upgrade successfully.

Take a Backup​

Before you attempt any software update, take a full backup of ...

Continue reading the Original Blog Post.

 

[David Liddle]

This information is very helpful, Herr Lüken. One of the reasons I am waiting to install the upgrade on our system is that it is a migrated Askozia appliance purchased back in 2015. I can't be 100% certain that the upgrade will run perfectly, but your explanation of common upgrade failures increases my hopes. Nevertheless, in the worst-case scenario, I have a virtual machine ready to take a fresh installation of 3CX which can then import a backup saved daily to our file server.

 

[coreybrett81]

Could you explain how the system is restored to it's previous state if the upgrade fails? Are you using some type of file system snapshotting?

 

[Agathocles Prodromou]

Could you explain how the system is restored to it's previous state if the upgrade fails? Are you using some type of file system snapshotting?

Hi @coreybrett81 ,

A system may fail/exit during various stages of an upgrade. Depending on the stage, the changes done to a system may vary from none to many. The more the changes the more difficult it is to restore.

If no changes were made, meaning the script exited at the very beginning, then it is as if it never ran.

During an ordinary 3CX upgrade, 3CX is uninstalled but the data is still there. This allows for the next version to be installed and use the same data.

Having said that, there are cases where 3CX is uninstalled but the changes made to the system still allow the previous version (V16) to be reinstalled/restored. This may not be possible to do at all stages, however when it is, we perform it.

It should be noted that the purpose of this is to give admins time to setup a new machine and properly migrate to V18. A system that was reverted to V16 by the upgrade script shouldn't be considered as if it was restored from a snapshot. (A snapshot is something an admin needs to take before initiating any upgrade process)

 

[ivan.antic]

@ all 3CX'ers

I can confirm that the upgrade script, using Upgrade button, work perfectly fine with previously manually installed v16.

That is:
Default Debian 9.
Configuration using ansible playbook with setupconfig.xml.
Additional tweaks (nginx + nrpe ...)
Upgrade using button.
All looks good in /tmp/UPDATE.log
::SUCCESS::
====== [10:19:36] Finish: New Debian version is 10
====== [10:19:36] Finish: Rebooting now

Voilla ... V18

cya

 

[FridayIT]

Good day everyone,

My two biggest tips I have used for the upgrade

1. Read the log file, 3cx has a pretty good upgrade log and it will usually at least point you in the right direction.
- I like to open two ssh sessions and run the htop command with one, and use the other one to "watch" a live feed of the log file so I can see the progress.

2. Snapshot before upgrading.
- Make sure you have a backup plan in place in case the update fails. our update failed and it left the install in "limbo" due to the perticular way it failed (nginx failed as it was pulling from a different location for the certificate).

I want to empisize, the update failure was *not* caused by 3cx, it was a configuration issue on our end.

 

[F.Haeberle]

Today, i wanted to upgrade my linux distribution - but I stopped, because I was not sure if there would be problems with 3cx. We are working every day with 3cx and I want to be sure, that I don't have a problem.
Is it there a danger with 3cx updating Linux with apt-get update and apt-get upgrade?

Installing Curl on Debian​

1. Open the terminal application
2. Update system, run: sudo apt update && sudo apt upgrade
3. Install curl on Debian, execute: sudo apt install curl
4. Test curl

 

[ChrisC_3CX]

@F.Haeberle

The blog post specifically refers to the upgrade from 3CX v16 (Debian 9) to 3CX v18 (Debian 10). Upgrading the OS or packages on the OS is not recommended and may cause issues with 3CX such as "breaking" dependencies.
That said, upgrades on a Linux 3CX Host should only be performed via the 3CX Management Console. If you want to have Debian security updates installed you can enable 3CX Automatic updates in the Management Console's "Updates" section.

Note: Also bear in mind that no third party packages should be installed either (other than what's included in the 3CX ISO) as the host machine should be dedicated to 3CX.

 

[timmbo]

This information is very helpful, Herr Lüken. One of the reasons I am waiting to install the upgrade on our system is that it is a migrated Askozia appliance purchased back in 2015. I can't be 100% certain that the upgrade will run perfectly, but your explanation of common upgrade failures increases my hopes. Nevertheless, in the worst-case scenario, I have a virtual machine ready to take a fresh installation of 3CX which can then import a backup saved daily to our file server.

Hi David

The Silver 1 Space Hardware(Newer-one) from Askozia won't work with the Upgrade, the Network-Chip is not recognized, so not Networking Possible.
I checked two of them.

Bye
Timm

 

[FridayIT]

Is it there a danger with 3cx updating Linux with apt-get update and apt-get upgrade?

You don't want to use apt upgrade, there was some issues about a year back where users who were manually updating debian broke some packages that were used for push notifications.

Best way to think of it is consider 3cx as an "appliance VM" and you want to keep it as clean as possible.

 

[SteveITS]

You don't want to use apt upgrade

True.
I am replying to note for the OP that if one has automatic updates turned on, 3CX will install OS security updates.
If automatic updates is off, they are not installed even when a 3CX update is installed manually.

 

[QueBall]

A problem I would add. After upgrade, if you get a different IP address it likely means the VM was set for DHCP and the upgrade changes from MAC address (hardware address) in Debian 9 to a DUID based identifier in Debian 10

To solve this you either can update your dhcp reservation to use the new duid based client identifier, or you can change the dhclient.conf file to change back to the old behaviour.
Force Debian 10 to use MAC address for DHCP
https://superuser.com/questions/1553094/force-to-use-mac-address-as-dhcp-client-id-on-debian-buster

echo "send dhcp-client-identifier = hardware;" >>/etc/dhcp/dhclient.conf
rm /var/lib/dhcp/*
systemctl restart networking

That should get your server IP address back to where you expected.

 

[Oliver Broad]

As an alternative to trying to update-in-place I'd like to know if it is practical to swap the hard drive for a blank one, install the new iso, then restore a backup onto the fresh install.

Alternatively I could take the opportunity to migrate to a new box, possibly a PI. Either way the old HD would be left alone, meaning if things don't migrate I could revert quickly by just booting up the old install.

For clarification I'm asking because I haven't actually done a "restore from backup", though I belive I've confirmed the integrity of the backup as it can be decompressed.

 

[N_G]

@Oliver Broad No its not practical. Backup and restore is much easier way to go.....

 

[elounda]

We are on Version 18 and we are running a cloud instance.

After the automated upgrade we are unable to see Bridged 3CX instances.

• Under Sip Trunks, Registration from each 3CX instance is successful.
• I can even make calls to remote pbx instances

I cannot see them (the extensions of the remote pbx) and they cannot see me

All other instances are running 3CX version 16 in Windows Operating Systems (there is no V18 upgrade for them at the moment, and they cannot upgrade)

3CX Is there any solution?

 

[CentrexJ]

We are on Version 18 and we are running a cloud instance.

After the automated upgrade we are unable to see Bridged 3CX instances.

• Under Sip Trunks, Registration from each 3CX instance is successful.
• I can even make calls to remote pbx instances

I cannot see them (the extensions of the remote pbx) and they cannot see me

All other instances are running 3CX version 16 in Windows Operating Systems (there is no V18 upgrade for them at the moment, and they cannot upgrade)

3CX Is there any solution?

It sounds like your v18 doesn't think it's licensed with Pro or Enterprise (bridging doesn't work on standard/free). On 3CX management console dashboard, license, hit refresh. Then if it still doesn't work restart the vm or restart the 3cx services.

 

[NickD_3CX]

We are on Version 18 and we are running a cloud instance.

After the automated upgrade we are unable to see Bridged 3CX instances.

• Under Sip Trunks, Registration from each 3CX instance is successful.
• I can even make calls to remote pbx instances

I cannot see them (the extensions of the remote pbx) and they cannot see me

All other instances are running 3CX version 16 in Windows Operating Systems (there is no V18 upgrade for them at the moment, and they cannot upgrade)

3CX Is there any solution?

Hi!

Yes, the first things you need to do is upgrade all other 3CX installations to V18 as well. With bridged systems, all of them need to be at least on the same major release, preferably also the same Update.

After doing this, try what @CentrexJ suggested, jsut refersh the license key information.

 

[elounda]

Hi!

Yes, the first things you need to do is upgrade all other 3CX installations to V18 as well. With bridged systems, all of them need to be at least on the same major release, preferably also the same Update.

After doing this, try what @CentrexJ suggested, jsut refersh the license key information.

How I am supposed to do that..... all other instances are in Windows Version (does 3CX has Windows Upgrade Available?)
And why they auto upgrade since they are still in progress with windows Versions.

Why does 3CX try to prove something out of nothing, give the upgrade and let companies decide if they want it or not....

 

[elounda]

How I am supposed to do that..... all other instances are in Windows Version (does 3CX has Windows Upgrade Available?)
And why they auto upgrade since they are still in progress with windows Versions.

Why does 3CX try to prove something out of nothing, give the upgrade and let companies decide if they want it or not....

is there a way to roll back the auto upgrade? , is there any automate downgrade?

 

[CentrexJ]

is there a way to roll back the auto upgrade? , is there any automate downgrade?

3CX doesn't auto upgrade from v16 to v18, not sure what you are referring to.

Please try the license refresh as suggested, it might be sufficient to get you operational for now.