Security & Memory Hotfix Available for V18 Update 3

[NickD_3CX]

View attachment 28869
apt dist-upgrade delivers the same issue.
Proxmox works fine with KVM the last weeks. So it was a kind of surprise to get such an issue.

And yes, bare metal is nice, but if you have a crashed phonesystem it took much longer to "roll it back" to working.
So it was just 1 click and 2 minutes. Love virtualisation for making life much easier... most of the times. ;-)

NGINX status bevor AND after crashed Update:
View attachment 28870

KVM:
View attachment 28871
So KVM is maybe not the best choise, but 3CX knows about... ;-)

The nginx warning is just that, a warning which does not cause any trouble at all, in fact we have recently double-checked this and there is no error in the config.

I know that a lot of users use Proxmox environment for their 3CX, and I have read most of the arguments about it running on KVM, etc.
The fact of the matter is, we don't test it so we can't guarantee it will run smoothly.

We advise using only tested/supported VM environments.

 

[rasim]

thanks @NickD_3CX , we will roll out hotfix to all customers as soon as possible and keep monitoring memory consumption

PS. Just rolled this update out along with SBC 18.1.36 update to ~40 customer instances. Will post again with feedback after EOBD tomorrow.

 

[IT Consulting Lucke]

I know that a lot of users use Proxmox environment for their 3CX, and I have read most of the arguments about it running on KVM, etc.

Maybe it would be nice to support a lot of Proxmox user as well?

Some Proxmox user I know did this Update without an issue. I'll see what the differences are...

 

[brett.buchanan]

System crashed.
3CX on Proxmox 7.1.10 PVE in VM:
Ran Update, SSH still alive but Android App could not connect. Login to Admin Panel ended in Server Error!

Hi Lucke,

I have just had the exact same issue but this customer's instance was hosted on AWS.

Long story short, the 3CX Management Console service would not start and just timeout trying to start. After increasing the time-out timer to 600s it still would timeout. The quickest way to resolve this issue was to restore the backup directly to a new v18 instance. This instance was a semi-large instance with ~150 extensions.

 

[IT Consulting Lucke]

I'll have an eye on this update, disable the auto update and will see when the customer has time to deploy a new instance and restore the backupfile. But there must be a reason why the update crashs. It's good for everyone that we report those things.
Thanks for your feedback regarding AWS.

 

[jofer]

I have the same problem on Windows server 2016, after this update. The management console service is not starting. This is a very large system, no home users can work...

 

[NickD_3CX]

I have the same problem on Windows server 2016, after this update. The management console service is not starting. This is a very large system, no home users can work...

RDP onto the server, open "Services" and locate the "3CX PhoneSystem 01 Management Console" service.
Can you start it manually?

Also, do you happen to have any AV software installed on the 3CX Server? In the past we have seen cases where if such software is installed and the paths that are mentioned here have not been excluded, some files may have been 'blocked' by the AV during an upgrade.

 

[jofer]

RDP onto the server, open "Services" and locate the "3CX PhoneSystem 01 Management Console" service.
Can you start it manually?

Also, do you happen to have any AV software installed on the 3CX Server? In the past we have seen cases where if such software is installed and the paths that are mentioned here have not been excluded, some files may have been 'blocked' by the AV during an upgrade.

Hi Nick
Yes we have SEP running on the server, this is a requirement of the customer. It has always been running on this server.
The service does not start even manually, I have been trying hours now.

 

[NickD_3CX]

Hi Nick
Yes we have SEP running on the server, this is a requirement of the customer. It has always been running on this server.
The service does not start even manually, I have been trying hours now.

Uninstall the AV software, reboot the 3CX Server and send me a PM once you've done this to see if there is anything we can do.

 

[IT Consulting Lucke]

This is why virtualisation is so sexy. Rollback in seconds...

 

[Sam2020#]

Dear All,
Our 3cx system got breached yesterday night. they actually access the admin console and collected all the extension's passwords and did a lot of calls. I think they got the admin access from the security vulnerability because no one has the admin password except me. so I would recommend everyone to update. i would love a response from any expert if what I said is possible and make sense.
thanks

 

[Pierre_3CX]

Dear All,
Our 3cx system got breached yesterday night. they actually access the admin console and collected all the extension's passwords and did a lot of calls. I think they got the admin access from the security vulnerability because no one has the admin password except me. so I would recommend everyone to update. i would love a response from any expert if what I said is possible and make sense.
thanks

Hello, sorry to hear of this, please reach us asap on [email protected] to review traces of the incident.
Note if your system runs on Linux it isn't related to the security vulnerability.

 

[IT Consulting Lucke]

So to be fair:
I had 3CX running in a Proxmox VM with CheckMK Monitoring. Works great but:

3CX installed, with CheckMK Agent -> update fails
3CX installed, CheckMK Agent removed, uninstalled and in Systemd it was not listed anymore -> update fails
3CX reinstalled, backup data restored -> update runs.

Thank you 3CX for supporting KVM virtualisation. Saved hours of work!

 

[Minime]

Dear All,
Our 3cx system got breached yesterday night. they actually access the admin console and collected all the extension's passwords and did a lot of calls. I think they got the admin access from the security vulnerability because no one has the admin password except me. so I would recommend everyone to update. i would love a response from any expert if what I said is possible and make sense.
thanks

Hello,
we have a 3cx v18 running on Debian, so on the lucky side.
But the are entries in our Mainfirewalls log with connections from a Network in China to Port 5001 on the 3cx System.
We don't have staff in China so I guess the exploid is in the wild.

 

[Pierre_3CX]

Hello,
we have a 3cx v18 running on Debian, so on the lucky side.
But the are entries in our Mainfirewalls log with connections from a Network in China to Port 5001 on the 3cx System.
We don't have staff in China so I guess the exploid is in the wild.

I wouldn't jump into such conclusion as we constantly see scans for weak passwords whether on web or SIP ports, and that's not something new.

As long as your credentials are strong there is no risk as the blacklist will trigger after few repeated failed authentications attempts.
Ensure having the 3CX Global Blacklist feature enabled in Security / Anti-hacking so that traffic from known scanners is dropped directly.

 

[Minime]

I wouldn't jump into such conclusion as we constantly see scans for weak passwords whether on web or SIP ports, and that's not something new.

As long as your credentials are strong there is no risk as the blacklist will trigger after few repeated failed authentications attempts.
Ensure having the 3CX Global Blacklist feature enabled in Security / Anti-hacking so that traffic from known scanners is dropped directly.

But your StefanW(CEO) wrote "Do you run 3CX on Windows? Then update!"
This did not sound like inbuild Security can protect against this.
Strong credentials dont help if you can bypass authentication, or watever the security-fix is for.

 

[Pierre_3CX]

But your StefanW(CEO) wrote "Do you run 3CX on Windows? Then update!"
This did not sound like inbuild Security can protect against this.
Strong credentials dont help if you can bypass authentication, or watever the security-fix is for.

I'll PM you so we can check logs (depending verbosity at the time of the scan). That said I wouldn't be surprised if that's a standard scan sweep for weak credentials as our Global Blacklist litterally counts 100,000s IP entries and keeps growing on daily basis with the wide majority being from such "dumb" scans.

 

[SteveITS]

Maybe it would be nice to support a lot of Proxmox user as well?

We don't use Proxmox but I was surprised to learn here a while back that only some KVM installs are officially supported. I had suggested 3CX list those on their spec page but it isn't listed yet.

 

[Michael Menor]

Is there a CVE related to this vulnerability?

 

[StefanW]

Is there a CVE related to this vulnerability?

No, not at the moment. After the 31st March, however, this will change...