Update Failing (SaltStack issue?)

[ip25]

Evening All,
We've got a few instances that won't update with the below logs, all running on Debian and deployed using the PBX Express tool. Trying to get the latest update rollup...


Hit:1 http://downloads-global.3cx.com/downloads/debian stretch-testing InRelease
Hit:2 http://security.debian.org/debian-security stretch/updates InRelease
Ign:3 http://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch
InRelease
Err:4 http://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch Release
404 Not Found [IP: 13.224.67.36 80]
Hit:5 http://downloads-global.3cx.com/downloads/debian stretch InRelease
Ign:6 http://cdn-aws.deb.debian.org/debian stretch InRelease
Hit:7 http://cdn-aws.deb.debian.org/debian stretch-updates InRelease
Hit:8 http://cdn-aws.deb.debian.org/debian stretch Release
Reading package lists...
E: The repository 'http://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0
stretch Release' does no longer have a Release file.


Any thoughts?

Kind Regards,
Nathan

 

[dredd0606]

Hi @tonybgi

You will absolutely need SSH access to resolve this.

Can I safely assume 3cx will be fixing this? Or do I need to ssh into all the instances I support to fix this?

 

[JohnS_3CX]

Hi @dredd0606

Unfortunately no - if any instances point towards a dead repository then the update mechanism will stop.

This means that the system cannot receive the update that fixes this.

If you have any instances that produce this error, you will have to run that command on them so they can proceed with the updates.

 

[Lee Cramman]

You may use this script to fix the OS and update the instance manager installer

wget -O - https://downloads.3cx.com/downloads/misc/debianpatches/9/16/0/5/20200619-os-update.sh | bash

If you're Google hosted you'll need elevation to run this script. By default, no password is set for so su / sudo doesn't work.

You just need to "sudo passwd" to set one and then run the script as su.

 

[Lee Cramman]

Might be an idea for this thread to be a temporary sticky.

 

[PeterJL]

Let me get this right... So, the system has got itself into a position where I've now got to log in to each one and fix it.

Good 'test before release' process! Can't say I'm happy about this!

 

[Lee Cramman]

Let me get this right... So, the system has got itself into a position where I've now got to log in to each one and fix it.

Good 'test before release' process! Can't say I'm happy about this!

Not great... However, not all instances are having this issue so hopefully just the odd one of yours needs this fix.

 

[JohnS_3CX]

Sorry about that Peter, the repository was taken down which is something we do not control and this is something that happened after the release was already out hence it would not show up in testing unfortunately.

The script above, along with our next update, will ensure that this condition cannot cannot stop the updates again in the future

 

[PeterJL]

Thanks... checking does seem to show that not all have been affected, hopefully, just one or two, fingers crossed.

 

[Brian Birmingham]

Would the procedure be the same if we are hosted in Google cloud? I'm a little lost as there is no script. Just a download button and it runs on its own.

 

[AZ-Brad]

Anyway for 3cx to push a update that will fix this. I have 40 or so systems with same complaint. Total PITA when its Google Cloud or On Prem. lots of steps to get connected.

 

[AZ-Brad]

Would the procedure be the same if we are hosted in Google cloud? I'm a little lost as there is no script. Just a download button and it runs on its own.

You'll have to go into the Cloud console, hit Compute Engine, you'll see your instance that should say something like gc-custom-us-xxxxxxxxyyyyyyyy show you the internal IP and then External IP and next to that it says SSH. Click on SSH. That will give you console access.

 

[JohnS_3CX]

Hi @Brian Birmingham

As Brad said you can log in and launch an SSH session from here


You can elevate by typing sudo su and enter, then paste the command in and hit enter again.
Once done you can start the update from your management console and it should proceed smoothly.

 

[JohnS_3CX]

Anyway for 3cx to push a update that will fix this. I have 40 or so systems with same complaint. Total PITA when its Google Cloud or On Prem. lots of steps to get connected.

Unfortunately no, the problem is that you cannot download updates in the first place so even if we release one you cannot apply it without doing this step first.

The next update you will receive though, will ensure that this impasse cannot happen in the future, as our updater will be able to ignore soft errors and only stop on critical hard errors.

 

[Rolteus]

Hi @tonybgi

You will absolutely need SSH access to resolve this.

I have run this command via ssh and the result claimed that i would not be able to update my server or use instance manager to do it. i edited saltstack.list against my own best judgement according to instructions from users here to no avail. my system says the update is outdated. my update log says repositories are unavailable. ive done everything i can here is there something im missing? not sure why a routine update has to be such a hassle. would appreciate some help with this issue i did not cause and have no idea how to resolve. thank you.

 

[Rolteus]

Unfortunately no, the problem is that you cannot download updates in the first place so even if we release one you cannot apply it without doing this step first.

The next update you will receive though, will ensure that this impasse cannot happen in the future, as our updater will be able to ignore soft errors and only stop on critical hard errors.

is there a place where i can get up to date information on this issue? when is the next update? i had a problem in the past the pbx was not updated i had to rebuild. im really trying to keep up with everything but this is impossible to follow. can you tell me what's going on with updates, when will this current update issue be resolved and when will a proper functional update be applied? can i still use the automatic updater?

Thank you
Tom

 

[JohnS_3CX]

Hi Tom,

We really need to see the output of the command once you run it so we can better understand the situation on your machine.

Also tell us what version your PBX is currently (appears on your 3CX dashboard on the rightmost column)

 

[Rolteus]

Hi Tom,

We really need to see the output of the command once you run it so we can better understand the situation on your machine.

Also tell us what version your PBX is currently (appears on your 3CX dashboard on the rightmost column)

im assuming the command you mean is:
wget -O - https://downloads.3cx.com/downloads/misc/debianpatches/9/16/0/5/20200619-os-update.sh | bash

when i attempted to run this command again it ran successfully. i now have this new outdated update that my system will not allow me to apply. see screenshot.

 

[JohnS_3CX]

Hi Tom,

If you reached 16.0.5.622 then you are on the latest stable version and there is no further need for updates.

The other update you see is a beta preview that you can safely ignore.
When the next stable release comes out, it will say "Release" in the place of "Beta" and that one you can upgrade to safely when the time comes.


Looks like you are all up to date now

 

[Phill.]

thanks,

ssh to instance

sudo su
wget -O - https://downloads.3cx.com/downloads/misc/debianpatches/9/16/0/5/20200619-os-update.sh | bash

then update via GUI did the trick

 

[complit]

confirmed solution - Thanks