3CX database upgrade failed

[peterdavie]

Our 3CX system failed to upgrade cleanly from version 18.0.5.418 to version 18.0.6.908 and 3CX is now inoperable and refuses to allow logins to the Management Console ("Server Error"). The following error is being reported during the upgrade process (this output is extracted from .../Logs/Update.log):

Updating from Version 18.0.6.908
Loading global scripts...
Loading instance scripts...
Loading instances...
CurrentDbVersion=542
Updating global DB tables...
Applying script for all instance tables
Loading instance parameters from phonesystem_mastertable
Updating instance DB tables...
Replacing parameters
Npgsql.PostgresException (0x80004005): 42601: syntax error at end of input
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteScalar(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteScalar()
   at Updater2Db.UpdateInfoExtractor.ExecuteScript(String sqlQuery, String transactionName, Exception& err)
  Exception data:
    Severity: ERROR
    SqlState: 42601
    MessageText: syntax error at end of input
    Position: 36
    File: scan.l
    Line: 1119
    Routine: scanner_yyerror
Npgsql.PostgresException (0x80004005): 42601: syntax error at end of input
   at Npgsql.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteScalar(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteScalar()
   at Updater2Db.UpdateInfoExtractor.ExecuteScript(String sqlQuery, String transactionName, Exception& err)
  Exception data:
    Severity: ERROR
    SqlState: 42601
    MessageText: syntax error at end of input
    Position: 36
    File: scan.l
    Line: 1119
    Routine: scanner_yyerror

Any help or suggestions are welcome.

 

[peterdavie]

That worked for us.

Thank you very much for your help!!

 

[Rich.Hopkins]

Got the same here, what worked?

 

[adconsulting]

Same problem, what's the workaround?

 

[adconsulting]

It's very urgent. We have companies down.

 

[Rich.Hopkins]

Restoring from Friday's backup was successful. Can log into Management Console, but clients still unable to log in either via web client or phone app.

 

[adconsulting]

We don't have any backup do you know if 3cx does it before upgrade?

 

[JohnS_3CX]

@adconsulting and @Rich.Hopkins send me a private message with the contents of your /var/log/postgress logs

 

[adconsulting]

Hi everyone,
I was able to solve the problem in Linux installation. The process that stuck everything is dpkg that is executed during the upgrade.
If you inspect trough shell using ps aux | grep dpkg you can see a particular process called

/usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/grub-pc.postinst configure 2.02+dfsg1-20+deb10u4

Now you can kill it using kill (PID discovered by the command before eg 14860).
After that the upgrade process will continue and you will see again the process before. Do again the procedure for killing the process and it will continue to update. After 1 minutes the console will be avaiable again.

NOTE: This is will not upgrade the system but is usable to restore fully functional the system.
NOTE2: This work ONLY if you haven't reboot the system.

 

[Marc Reutenauer]

Hi everyone,

I also solve the problem, I just start manually all this services :
service 3CXCfgServ01 start
service 3CXPhoneSystem01 start
service 3CXSystemService01 start
service 3CXEventNotificationManager start
service 3CXQueueManager01 start
service 3CXTunnel01 start
service 3CXIVR01 start
service 3CXMediaServer start
service 3CXCfgServ01 start
service 3CXPhoneSystemMC01 start
service nginx start

But the problem appears again if you reboot your system.
I tried a new update that failed again.

 

[Rich.Hopkins]

Problem I have is the DB service won't start:

When you try to restart it, the MC comes up with Server Error again. So have to SHH to the server and start services there. to make the MC available again.

 

[Rich.Hopkins]

Getting this message I've only noticed this morning:
Could not having this set up yet be a cause?

 

[Charles_3CX]

For those who still have the issue and have installed the system using a supported method using the 3cx ISO on-premise or deployed the system via 3cx wizard to a supported cloud provide, listed here: https://www.3cx.com/docs/cloud-pbx-providers/ and have not made any modifications to the system, adding any 3rd party tools, etc.. Kindly open a ticket from your 3cxc portal for us to check for you.

For any others that fall out of the above pre-requests, we can also suggest taking the latest 3cx backup and re-deploying their instance to a 3CX dedicated hosted machine.

 

[Charles_3CX]

As a side note, we had had similar cases in which, after investigation, had determined the issue was related to the 3cx database still at an older version than that which it should have been when the OS was updated from Debian 9 - 10 and was caused by manually updating the OS.

In these cases, we recommend using your latest 3cx backup to spin up a new machine using a supported method, ISO for on-prem, 3cx Wizard from portal.3cx.com for cloud-based or supported marketplace deployment restoring the backup.

 

[JvZ]

Same problem here. Charles is right at my side of installations. There is a current Postgres 9 version 3CX database live, and there is 3CX Postgres11 version installed. Unfortunately My 3CX 18.5 is running on the (old) Postgres 9 database i.s.o. the version 11.
Is there a way to change the current running 3CX instance to point to the new version 11 database?

 

[Charles_3CX]

Same problem here. Charles is right at my side of installations. There is a current Postgres 9 version 3CX database live, and there is 3CX Postgres11 version installed. Unfortunately My 3CX 18.5 is running on the (old) Postgres 9 database i.s.o. the version 11.
Is there a way to change the current running 3CX instance to point to the new version 11 database?

I’m afraid not, the only option is to take a backup and re deploy to a new machine as per my previous reply.

Avoid any manual updates to ensue this issue does not re occur in the future.

 

[nanosistemi]

Hi all, IMHO the problem is only the wrong upgrade of postgres cluster.
In my PBX, debian based, I fixed manually upgrading postgres from 9.6 to 11:

su - postgres -c 'reindexdb --all'
su - postgres -c 'pg_dropcluster --stop 11 main'
su - postgres -c 'pg_upgradecluster 9.6 main'
su - postgres -c 'pg_dropcluster --stop 9.6 main'
apt autoremove postgresql-9.6
apt autoclean
reboot

Bye

Davide De Bona
www.nanosistemi.com

 

[II-Felipe]

Hi all, IMHO the problem is only the wrong upgrade of postgres cluster.
In my PBX, debian based, I fixed manually upgrading postgres from 9.6 to 11:

su - postgres -c 'reindexdb --all'
su - postgres -c 'pg_dropcluster --stop 11 main'
su - postgres -c 'pg_upgradecluster 9.6 main'
su - postgres -c 'pg_dropcluster --stop 9.6 main'
apt autoremove postgresql-9.6
apt autoclean
reboot

Bye

Davide De Bona
www.nanosistemi.com

That worked!! You're a legend.

Thank you!!

 

[Techops]

Hi all, IMHO the problem is only the wrong upgrade of postgres cluster.
In my PBX, debian based, I fixed manually upgrading postgres from 9.6 to 11:

su - postgres -c 'reindexdb --all'
su - postgres -c 'pg_dropcluster --stop 11 main'
su - postgres -c 'pg_upgradecluster 9.6 main'
su - postgres -c 'pg_dropcluster --stop 9.6 main'
apt autoremove postgresql-9.6
apt autoclean
reboot

Bye

Davide De Bona
www.nanosistemi.com

Felipe is right, this worked for us too, thank you!!