Update 1 install with anti virus active

[saskvark]

Just checking if anyone else got "Server Error" or unable to connect when logging into Management Console on v16 after Update 1.

It looks like on May 19 at 7pm some files were updated in the bin directory. The next day the phones were dead.

Although most services restarted after a second reboot, so at least the system is operational, the Mgmt Console service is stopped and times out when attempting to restart. Also, no automatic backups are being created and BackupCmd and RestoreCmd no longer work from the command line.

Short of a reinstall or downgrade to v15.5, anybody have any suggestions? Or run into the same thing?

 

[AWS2P]

Hi,
I assume you use a W10 pro PC with enough hardware capabilities for your use.
I never had this behavior, is 3CX set with automatic update on?
Do you have WU also running on computer? if so, did you set to apply update at night and not in working hours?

Do you have something registred in eventviewer at suspected time problem occurs ?

have you done a computer reboot since this happened?

For now is your 3CX running properly?

 

[saskvark]

Thanks for responding, aws2p. Sorry for the delayed response (very different timezone).

The host computer is W10 Pro 64-bit, a somewhat older machine but adequate (i5-2400, 4GB). Windows Updates are set for after hours, of course, and I'm assuming 3CX auto update must have been on (can't check without the mgmt console AFAIK) because there are 15 new 3CX files in the Instance1 bin folder timestamped for May 19 at 7:00pm (and a new codec folder). They are version 16.0.1.273 versus other files being the original version 16.0.0.1581.

And then at 7:01pm same day, nginx and its child processes shut down according to Windows Applications log, followed by PostGreSQL announcing its connections had been closed by the host.

Nothing else out of the ordinary in Event Viewer. After two reboots the system is operational and nginx is running, but Mgmt Console service will not start. So the phones work but no console. Also running BackupCmd or RestoreCmd from the command line fails with an unhandled exception due to a mismatch on the 3CXNetBackupandrestore dll. So my hands are a bit tied other than a reinstall, unless anyone as any other ideas?

 

[JohnS_3CX]

Hi @saskvark

It sounds like the update may have not been successful or was interrupted. You can check under C:\Program Files\3CX Phone System\Bin and read the files Updater1.txt and Updater2.txt

If you have antivirus running on the machine, make sure it is disabled during the update, some of them may interfere and block the update half way through, leaving the phone system disabled until you restore or reboot.

You will need a good backup from before the event occured, hopefully you kept a recent one if you need to restore.

 

[saskvark]

Thanks, JohnS_3CX. I had a look at the Updater logs and all looks fine except Updater2 ends abruptly at:

2019/05/19 19:01:29.870|0001| Service 3CX PhoneSystem Database Server is Stopped now
2019/05/19 19

But there is nothing in the Windows events indicating any interrupting event, so who knows. Anyhow I'll uninstall this weekend and reinstall from a backup.

Thanks for the input!

 

[JohnS_3CX]

Hi @saskvark

Yes, the abrupt ending of the file is a clear sign that the update failed for some reason. Regardless of the cause the installation is now compromised and it is best to reinstall to ensure you have a healthy system.

Once you restore from backup and rebuild the system, I would recommend to disable automatic updates, and when the next service pack arrives, run a quick backup and then launch the update manually from the MC.

If the failure occurs again then then it might point to something on your system that is interfering with your updates and you will have chance to address it. It could be an antivirus, or it could be that windows updates were running at the time, or some other third cause that's still unknown.

 

[Shawn]

check to see if system restore has a restore point prior to the upgrade. that will allow you to revert back to a point in time prior to the update.

food for thought for the future. i would install the veeam free end point backup or equivalent. put another hard drive in the machine and back up the entire machine (system volume and data) to the 2nd drive using the scheduled feature of the software. the backup is designed for bare metal recovery which will allow you to restore to dissimilar hardware if you have a catastrophic failure. in this case, you could have simply picked a point in time and reverted back to then, as if nothing happened. the software is free and works well. make sure you create a bootable recovery iso.

you could also virtualize the installation and use scheduled snapshots which make reverting back much quicker and easier.

 

[saskvark]

Thanks again for the input, folks. Shawn, I wasn't aware of veeam, it sounds good. Fortunately we had good backups via 3CX management console so once the software was reinstalled I could restore the system fairly easily.

FWIW, we did exclude the 3CX system folders from the antivirus, but the install failed again as soon as it tried to update itself, and at exactly the same point. So far now auto updates are off.

 

[N_G]

This is clearly anti virus corrupting your system, absolutely nothing to do with update 1