V16 Upgrade issues causing console to crash

[Bonsai_UK]

Hi All - Need some help please hoping you all can offer some advice.

Some background info:

Large group of Colleges in the UK with 4000 users across 8 sites linked with point to point fibre lines and MPLS

Mix of around 2800 Yealink handsets and Windows Desktop Clients + mobile apps

Windows Deployment on Premise using VMWare

8 CPU Cores / 16GB RAM (now 32GB RAM) / 10 Gig NIC

Enterprise 3CX licence supported by a major platinum 3CX partner in the UK

• After happily running version 15.5 for the last couple of years we decide (pushed actually due to 3CX pulling the mobile apps) we wanted to upgrade to version 16 update 5 and move to the cloud. Please note at this time everyone was working on site and not from home yet.
• 
  
• Reseller upgrades to version 16 first on premise runs fine for half a day and then fails. Unable to access the 3CX console error saying IP mismatch and certificate errors in the logs. We revert back to 15.5 after seeing the new server installed some windows updates which was put down to causing the issue as well as certificate errors.
• 
  
• Reseller says we will move straight to the cloud (Google and Linux setup). Reseller sets up the new server in the cloud tries to get it working similar issues and we revert back to 15.5 on premise and will talk to 3CX who said it was an issue on Linux release.
• Along comes to corona virus and we urgently need users to be able to use the mobile apps. Reseller agrees to try again to move to the cloud to a windows deployment. Guess what move fails we are told due to the IPsec links flooding the console. They want to revert to 15.5 on premise but we need the upgrade.
• 
  
• After discussing how urgently we need to get v16 working we agree to deploy a new server fully updated with windows updates. Reseller carries out the install and restore of our database. Eventually it comes up and seems successful.

Fantastic I thought this will get us through the storm for now and we can look at moving to the cloud again later. But…

• Few days later everyone is working from home then suddenly everything fails. Phones drop off everything goes dead and the console has the same IP mismatch error and logs show certificate errors. After getting to the bottom of being told the IP has been changed which we later confirmed it wasn’t the reseller started looking into it. At this point the system was down for 7 hours and work to fix it was going on into the evening. We left them to it and users started to log out of the systems. We are and soon told it was fixed and it was a permissions issue on the certificate folder.

Great finally fixed again. Or is it…

• Next morning system seems OK and stable for now. We ask support how to make a change to enable 3CX tunnel for Windows App users. We are shown how to do a bulk edit on some extensions. We go to make a small change on less than the 100 extensions 3CX allows at any one time. Bam system dies again and everyone loses access. Same errors in the console. For crying out loud here we go again I thought.
• 
  
• Partner is contacted and after some time they look into it again. At this point we been talking to senior managers within the partner company involved. We also decided to contact the channel partner of 3CX in the UK explaining this is now urgent and we needed their help.
• 
  
• Get an email last night saying it was back up and being worked on by the senior engineers of the reseller and 3CX. They still not found the issue blamed on a memory leak yet memory usage when observed never goes above 8GB out of the 32GB now provisioned on the server.

So my questions to you all are have you seen anything like this before? Do you have any suggestions on what to check and do next? It seems to be it cannot cope with the load of users being on the system and once they are all off it comes back. It clearly is very unstable if it cannot carry out a simple extension edit for a few users without falling over and will fail without any notice.

On a second note I will not be releasing who the reseller is just yet but I am keen to hear from UK resellers out there and suggestions as to ones you have found to offer excellent support.

Thanks all sorry for the long post but we are frustrated and we need to find a resolution to this urgently now.

 

[florink]

Emm, V16 Update 5 is in Alpha2...it shouldn't be installed on a production system! You should update to Update 4 for now.

 

[Bonsai_UK]

Sorry my bad it is V16 update 4. Well spotted.

 

[hogan71088]

With the move to the cloud are they using the 3cx SBC or vpn to the cloud instance?

 

[Bonsai_UK]

Hi Hogan

They are using multiple IPSec VPN's