upgrade from 18.0.7.132 to 18.0.8.912 = unable to login to management console -> server error

[kwen1x]

started upgrade from linux console (ssh, root login).. no errors.
Even phones are still working after upgrade.
BUT management console refuses login: server error.

Have rolled back from backup, redid upgrade.. same story.

Now what ?

 

[kwen1x]

Get:1 http://repo.3cx.com/3cx buster/main amd64 3cxpbx amd64 18.0.8.912 [128 MB]
Fetched 128 MB in 6s (21.8 MB/s)
Preconfiguring packages ...
(Reading database ... 39342 files and directories currently installed.)
Preparing to unpack .../3cxpbx_18.0.8.912_amd64.deb ...
Removed /etc/systemd/system/sysinit.target.wants/3CXFirewall.service.
Removed /etc/systemd/system/3CXEventNotificationManager.service.
Removed /etc/systemd/system/3CXAudioProvider01.service.
Removed /etc/systemd/system/3CXIVR01.service.
Removed /etc/systemd/system/3CXPhoneSystemMC01.service.
Removed /etc/systemd/system/3CXCfgServ01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXEventNotificationManager.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXAudioProvider01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXIVR01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXPhoneSystemMC01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXCfgServ01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXCallFlow01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXGatewayService.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXMediaServer.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXQueueManager01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXPhoneSystem01.service.
Removed /etc/systemd/system/multi-user.target.wants/3CXSystemService01.service.
Removed /etc/systemd/system/3CXCallFlow01.service.
Removed /etc/systemd/system/3CXGatewayService.service.
Removed /etc/systemd/system/3CXMediaServer.service.
Removed /etc/systemd/system/3CXQueueManager01.service.
Removed /etc/systemd/system/3CXPhoneSystem01.service.
Removed /etc/systemd/system/3CXSystemService01.service.
Unpacking 3cxpbx (18.0.8.912) over (18.0.7.312) ...
Setting up 3cxpbx (18.0.8.912) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Updating from Version 18.0.7.312
Loading global scripts...
Loading instance scripts...
Loading instances...
CurrentDbVersion=577
Updating global DB tables...
Applying script for all instance tables
Loading instance parameters from phonesystem_mastertable
Updating instance DB tables...
Replacing parameters
Adjusting timezone
Linux timezone file path = /usr/share/zoneinfo/Europe/Brussels
Configuring Linux timezone
Running /usr/bin/sudo /usr/sbin/3CXSetTimezone "Europe/Brussels"

Current default time zone: 'Europe/Brussels'
Local time is now: Mon Aug 7 13:18:33 CEST 2023.
Universal Time is now: Mon Aug 7 11:18:33 UTC 2023.



Created symlink /etc/systemd/system/sysinit.target.wants/3CXFirewall.service → /lib/systemd/system/3CXFirewall.service.
Created symlink /etc/systemd/system/3CXCfgServ01.service → /lib/systemd/system/3CXCfgServ01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXCfgServ01.service → /lib/systemd/system/3CXCfgServ01.service.


_____ _______ __
|__ // ____/ |/ /
/_ </ / | /
___/ / /___ / |
/____/\____//_/|_|

Welcome to the 3CX Configuration Tool
Help https://www.3cx.com/docs/manual/

Nginx configuration file has been successfully recreated
Created symlink /etc/systemd/system/3CXMediaServer.service → /lib/systemd/system/3CXMediaServer.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXMediaServer.service → /lib/systemd/system/3CXMediaServer.service.
Created symlink /etc/systemd/system/3CXPhoneSystem01.service → /lib/systemd/system/3CXPhoneSystem01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystem01.service → /lib/systemd/system/3CXPhoneSystem01.service.
Created symlink /etc/systemd/system/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.
Created symlink /etc/systemd/system/3CXGatewayService.service → /lib/systemd/system/3CXGatewayService.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXGatewayService.service → /lib/systemd/system/3CXGatewayService.service.
Created symlink /etc/systemd/system/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Successfully updated

 

[kwen1x]

I had created an owner account, as requested per previous updates.

I was unable to access management console (with root account), using these 'owner' credentials, after upgrading.
(restore from backup/snapshot)
I had to update my owner account using the 'rights' tab -> give access to management console (is DISABLED by default).
Once I had also this console access, I'm able to connect to management console using 'system owner' extension and password. The 'old' root account still works for ssh access, but no longer for management console access.

 

[kwen1x]

as in 'I told you so'... but forgot to tell you the whole story

 

[kwen1x]

@3CX: maybe check BEFORE upgrade that at least 1 'system owner' also has management console access ?

 

[JohnS_3CX]

started upgrade from linux console (ssh, root login).. no errors.

We don't support this, you should always upgrade from the management console so we can run all the necessary checks pre-upgrade.

as in 'I told you so'... but forgot to tell you the whole story

We have been publicizing this for a while now, but this is not going to lock you out even in this case.

The 'old' root account still works for ssh access, but no longer for management console access.

Debian root login is not the same as Management Console login. When you first installed you got to define the root admin of the 3CX management console, the account controlling everything in 3CX (which is not an extension). That should still be valid and can still log you in after update 8 so you can assign a system owner (which is a normal extension). So in total after installing a system you had 3 sets of credentials:

- Debian OS root account
- Management Console root admin account
- First extension account (which is now the system owner if you install fresh)

BUT management console refuses login: server error.

This is not related to any of the above at all. Wrong credentials will never generate "Server Error".

You probably tried to login too fast if I was to take a guess, since once the 3CXPhoneSystemMC01.service comes up it takes a few moments until its ready to accept credentials. Nothing serious to worry about, just give the system a few moments for the services to sync up (usually under a minute, depending on system size and hardware).

Hope this clears up any confusion.

 

[kwen1x]

We don't support this, you should always upgrade from the management console so we can run all the necessary checks pre-upgrade.


We have been publicizing this for a while now, but this is not going to lock you out even in this case.


Debian root login is not the same as Management Console login. When you first installed you got to define the root admin of the 3CX management console, the account controlling everything in 3CX (which is not an extension). That should still be valid and can still log you in after update 8 so you can assign a system owner (which is a normal extension). So in total after installing a system you had 3 sets of credentials:

- Debian OS root account
- Management Console root admin account
- First extension account (which is now the system owner if you install fresh)


This is not related to any of the above at all. Wrong credentials will never generate "Server Error".

You probably tried to login too fast if I was to take a guess, since once the 3CXPhoneSystemMC01.service comes up it takes a few moments until its ready to accept credentials. Nothing serious to worry about, just give the system a few moments for the services to sync up (usually under a minute, depending on system size and hardware).

Hope this clears up any confusion.

Thanks John,

but my findings are the follwoing..

Debian root login is not the same as Management Console login. When you first installed you got to define the root admin of the 3CX management console, the account controlling everything in 3CX (which is not an extension). That should still be valid and can still log you in after update 8 so you can assign a system owner (which is a normal extension).

The login of the root admin was no longer working, in contrast of what you are claiming.
When trying to logon to update 8 management console, I got 'server error'.

You probably tried to login too fast if I was to take a guess, since once the 3CXPhoneSystemMC01.service comes up it takes a few moments until its ready to accept credentials.

I have my doubts.. not that I'm not fast..
Upgrade has been done now for1 hour.. still 'server error' when trying to connect with (old) 'management console root admin account'

To summarize:
1. I had to rollback from backup/snapshot to previous version
2. I had to give my existing owner account also management console access, when logged in to that previous version with my 'management console root admin account'
3. re-run the update
4. now I can no longer connect with my 'management console root admin account', but only with my owner user/account with management console access.

As the problem is still there, I can give you anything you want to examine this further.

Question: could the password LENGTH play a role in this issue ?? e.g. new web interface truncates the password field to a length of xx characters ? My 'management console root admin account' password is longer then 32 characters...
(My owner account with console access only has 32 characters)

 

[JohnS_3CX]

Yes of course, we will examine further and get back to you shortly

 

[JohnS_3CX]

Can you please tell us what the password length was?

 

[kwen1x]

the length was (and is) 64 characters <grin>

 

[JohnS_3CX]

Thanks. Where was the 64 characters originally entered?

 

[kwen1x]

install was in 2020.. so difficult to say.. using the installer ?

 

[JohnS_3CX]

All our setup methods including the console (from 2020 at least if not earlier) have a hard limit of 50 so I'm not sure how you ended up with more characters than the field allows. This is one mystery that you can maybe tell us about one day

But yes, I can confirm that this is the reason you ran into this issue. I was able to replicate it when the password length is 64 characters (I forced 64).

Even if you didn't have snapshots though, you would still have the ability to reset the password (email sent to admin email if you enter the wrong password 3 times) so you would not have been left out in the cold

 

[kwen1x]

Hi John,

Thanks for your input/support/confirmation.

is there a sql query I can execute to verify/confirm the psw length of the 'management console root admin account' ?
Maybe the psw I enter gets truncated somewhere, as per your explanation (and I copy-paste it into login window).
I did not find my 'management console root admin account' in the 'user' table...

2nd issue:
I used the psw reset procedure you suggested to reset the psw of the 'management console root admin account', since I cannot reset this when logged in to management console using 'owner account', although this owner account has every possible right I can assign to it.
Here's the catch: the 'management console root admin account' and the owner account have the same email address. When I resetted the psw of the 'management console root admin account' (32 chars), I was unable to login using 'management console root admin account' and my owner account ??
So I guess the password reset procedure works on email address and when having 'double' email addresses in the system, the procedure fails,to update the psw correctly, somehow...
I need to test this further, but I'm busy on other issues next days.
I guess a check on unique email addresses should be considered ?

Snapshots saved me, again...

 

[kwen1x]

All our setup methods including the console (from 2020 at least if not earlier) have a hard limit of 50 so I'm not sure how you ended up with more characters than the field allows. This is one mystery that you can maybe tell us about one day

But yes, I can confirm that this is the reason you ran into this issue. I was able to replicate it when the password length is 64 characters (I forced 64).

Even if you didn't have snapshots though, you would still have the ability to reset the password (email sent to admin email if you enter the wrong password 3 times) so you would not have been left out in the cold

Hi John,

I forgot to reply you in my previous messages, so that you maybe did not see my reply from yesterday ?

Situation is now as follows:
'management console root admin account' has been reset to a 50 char value. Please note that the error message displays the criteria which are applicable for a new password, but does not mention the 'hard' limit of 50 when you try to enter more then that.
The password was reset in 2 steps:

1. password reset request on management console, which generates a unique link to a reset page. I used a short, simple password
   
   
2. after this reset, I was able to gain access to management console with my 'management console root admin account'. I then used the root credentials page to reset my password to something more complicated and 50 chars long.
   It's at this point that I got an error message when trying to copy/paste a 64 char password. It's this error message that does not mention the 50 chars limit...
   
   
3. the 'immediate' reset to a 50 chars password using step 1 did not work properly yesterday. See message here above. That's why I tried this 2 step approach. I have no explanation why the password reset approach did not work with 50 chars psw ? Maybe my (human) mistake ?

I guess the security screening that was at the base of this version, made that things were adapted/changed in this latest release, impacting the fact that my 64 chars password was no longer working after the upgrade ?!

 

[JohnS_3CX]

Hi,

You have me a bit confused here, but I think I got what you mean in the end. The short version is:

- Password reset by email: Only works when using unique emails otherwise you won't get the reset email in your inbox

- Password reset by username: Applies to root admin only, and does not care whether the email is unique

- Editing the Root Admin password via management console: if you enter more than 50 chars, you will get a warning (doesn't mention length)



So the gist of it is that upto 50 chars is fine, not quite sure how you had 64 originally as both U7 and U8 have the same limitation.

 

[kwen1x]

Hi,

You have me a bit confused here, but I think I got what you mean in the end. The short version is:

- Password reset by email: Only works when using unique emails otherwise you won't get the reset email in your inbox

No...
root admin and pbx owner have same email address, which I did not know/realize at the time of reset request.
I did the reset using email address, THINKING I was resetting my root admin account, but (I guess) I was also resetting 'owner account'.
The reset somehow failed using 50 chars password, leaving me unable to login with root admin and/or pbx owner.
So in the end (after restoring snapshot) I used the user name, just to be sure to update only 1 account and I used a short/simple password using the reset.

- Password reset by username: Applies to root admin only, and does not care whether the email is unique

yes, see reply above

- Editing the Root Admin password via management console: if you enter more than 50 chars, you will get a warning (doesn't mention length)

correct, length restriction is not mentioned and password is meeting all requirements that are mentioned, yet is it refused because it's too long..

So the gist of it is that upto 50 chars is fine, not quite sure how you had 64 originally as both U7 and U8 have the same limitation.

Still not sleeping very well because of this one :=)

 

[JohnS_3CX]

Yeah the last one is indeed a bit of a weird one, I was wondering whether you were using a password manager probably?

I'm not sure but sometimes I've seen password managers pasting stuff in a non standard way (maybe by manipulating the webpage code) so the validation fields may end up not getting triggered and the long string was entered without generating a warning. Just a hunch that might be worth testing..

 

[kwen1x]

Yeah the last one is indeed a bit of a weird one, I was wondering whether you were using a password manager probably?

I'm not sure but sometimes I've seen password managers pasting stuff in a non standard way (maybe by manipulating the webpage code) so the validation fields may end up not getting triggered and the long string was entered without generating a warning. Just a hunch that might be worth testing..

I was using windows version before moving to linux version in 2020.
Maybe before that password length was not enforced everywhere and when I restored my windows backup I had it running on my linux box ?

 

[JohnS_3CX]

Difficult to say, I'm sure at least on V16 it was enforced.