Threat alerts from SentinelOne for desktop update initiated from desktop client

[sfat3foldit]

My rep told me that 3CX's official stance is there is nothing malicious about their clients and concerned customers can use the WebClient. LOL, yeah ok..

 

[BGHIS]

Craziness. We are uninstalling all instances of the Desktop Application and are switching our users to the web client. Better safe than sorry and I would rather spend a few hours doing that as opposed to dealing with a data breach.

Hopefully 3CX is forthcoming with some useful information in the near future...

 

[alarayn]

https://www.bleepingcomputer.com/ne...ise-3cx-desktop-app-in-a-supply-chain-attack/
https://www.sentinelone.com/blog/sm...3cx-software-in-software-supply-chain-attack/

It's all over the news right now it seems.

 

[glipschitz_summit]

Sounds like their code base and, Keys, potentially CA and Code repo has been compromised.

 

[glipschitz_summit]

Doing some more digging on this...

It appears that the 3CX server itself does a background update which pulls the most recent Client to the local 3CX server.
From there, the client pulls the most recent update down.

Files are downloaded to the following locations

/var/lib/3cxpbx/Instance1/Data/Http/electron/osx/3CXDesktopApp-18.12.416.dmg
/var/lib/3cxpbx/Instance1/Data/Http/electron/windows/3CXDesktopApp-18.12.416-full.nupkg
/var/lib/3cxpbx/Instance1/Data/Http/electron/windows/3CXDesktopApp-18.12.416.msi

Looking at the datestamp, the file downloaded at

Mar 26 14:11 GMT+11

You can remove them by SSH to your server and using the following commands in the first instance.

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*

I am trying to find the daily file where it is triggering the update to get the client and to see what else it pulls to the server.

At this point, I would say their code base and CICD change has been compromised and this will be a very wide spread issue across every instance which has been rolled out.

Some acknowledgement from 3CX would be great as right now so that we can effectively communicate with our customers.

 

[glipschitz_summit]

It appears that the 3CX server runs this background command which lines up with the time the client was updated.

Start-Date: 2023-03-26 14:10:07
Commandline: apt-get -y --force-yes install 3cxpbx=18.0.7.312
Requested-By: phonesystem (999)
Upgrade: 3cxpbx:amd64 (18.0.6.908, 18.0.7.312)
End-Date: 2023-03-26 14:11:31

 

[bradgarrison]

I'm seeing reports on Google News about 3cx being hacked. Figured I would come here but it appears that is the last place to get updates.

 

[jbland]

3CX needs to put out a statement. Immediately.

 

[glipschitz_summit]

More logs which match the timestamp....
Looks likes it's a code base update which has pushed the update to their Repo.
Who knows what else they have done and other payloads they have dropped in the process here.

/var/lib/3cxpbx/Data/Logs/unattended-upgrade.log

Hit:1 http://repo.3cx.com/debian/1807 buster InRelease
Hit:2 http://repo.3cx.com/debian-security/1807 buster InRelease
Hit:3 http://repo.3cx.com/3cx buster-testing InRelease
Hit:4 http://repo.3cx.com/3cx buster InRelease
Reading package lists...
No updates installed.

 

[glipschitz_summit]

Looks like the update touches LOTS of files and directories.....


/var/lib# find 3cxpbx/ -newermt "2023-03-26 14:00:00" -not -newermt "2023-03-26 15:00:00+1"

3cxpbx/
3cxpbx/Instance1/Data/Ivr/Prompts
3cxpbx/Instance1/Data/Ivr/Prompts/Sets/8210986B-9412-497f-AD77-3A554F4A9BDB
3cxpbx/Instance1/Data/Ivr/Prompts/Sets/8210986B-9412-497f-AD77-3A554F4A9BDB/time
3cxpbx/Instance1/Data/Ivr/Prompts/Sets/8210986B-9412-497f-AD77-3A554F4A9BDB/numbers
3cxpbx/Instance1/Data/Ivr/Prompts/Hotel
3cxpbx/Instance1/Data/Http/Reports
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyExtensionStatisticReport_2603_zm70L3huk6rJbLLsaBDQ.html
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyQueueAnsweredcallsbywaittimeReport_2603_z8LZwDG5XfY002bZJcGY.html
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyRingGroupStatisticsReport_2603_rLbyA8ahxOwILaNl2SZr.html
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyQueuePerformanceReport_2603_G0iZ9j5mMubKHO3gCvfr.html
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyAbandonedQueueCalls_2603_6ENUYU0LR3SoJytseVUQ.html
3cxpbx/Instance1/Data/Http/Reports/Logo
3cxpbx/Instance1/Data/Http/Reports/xxxWeeklyDetailedQueueStatisticsReport_2603_rJWYZXcdcxF58ewFfgc6.html
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/fanvil_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/yealink_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/snom_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/aastra_phonebook.csv
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/gigaset_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/gs_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/cisco_phonebook.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/000000000000-directory.xml
3cxpbx/Instance1/Data/Http/Interface/provisioning/d85fcxvw6rk72h/logo
3cxpbx/Instance1/Data/Http/Interface/MyPhone
3cxpbx/Instance1/Data/Http/Templates/provider
3cxpbx/Instance1/Data/Http/Templates/phones
3cxpbx/Instance1/Data/Http/Templates/gateway
3cxpbx/Instance1/Data/Http/Templates/fxs
3cxpbx/Instance1/Data/Http/Templates/messaging
3cxpbx/Instance1/Data/Http/Templates/crm
3cxpbx/Instance1/Data/Http/electron
3cxpbx/Instance1/Data/Http/electron/windows
3cxpbx/Instance1/Data/Http/electron/windows/version.xml
3cxpbx/Instance1/Data/Http/electron/windows/3CXDesktopApp-18.12.416.msi
3cxpbx/Instance1/Data/Http/electron/windows/3CXDesktopApp-18.12.416-full.nupkg
3cxpbx/Instance1/Data/Http/electron/osx
3cxpbx/Instance1/Data/Http/electron/osx/3CX Desktop App-darwin-x64-18.12.416.zip
3cxpbx/Instance1/Data/Http/electron/osx/version.xml
3cxpbx/Instance1/Data/Http/electron/osx/3CXDesktopApp-18.12.416.dmg
3cxpbx/Instance1/Data/Logs/3CXWebMeeting.log
3cxpbx/Instance1/Data/Logs/3CXQueueManager.log
3cxpbx/Instance1/Data/Logs/3CXCallFlow.log
3cxpbx/Instance1/Data/Logs/3CXAudioProvider.2023-03-26.141059.log
3cxpbx/Instance1/Data/Logs/3CXQueueManager.track.log
3cxpbx/Instance1/Bin/Cert
3cxpbx/Instance1/Bin/Cert/root_cert_dt.pem
3cxpbx/Instance1/Bin/Cert/root_cert_voxtelesys.pem
3cxpbx/Instance1/Bin/Cert/root_cert_teamsdirectrouting.pem
3cxpbx/Instance1/Bin/Cert/root_cert_bouyguestelecom.pem
3cxpbx/Instance1/Bin/Cert/certificates.xml
3cxpbx/Instance1/Bin/Cert/Apple
3cxpbx/Instance1/Bin/Cert/root_cert_televoip.pem
3cxpbx/Data/Http/Reports
3cxpbx/Data/Http/wwwroot
3cxpbx/Data/Http/wwwroot/licenses
3cxpbx/Data/Http/wwwroot/livechat
3cxpbx/Data/Http/wwwroot/callus
3cxpbx/Data/Http/wwwroot/l10n
3cxpbx/Data/Http/wwwroot/webclient
3cxpbx/Data/Http/wwwroot/webclient/i18n
3cxpbx/Data/Http/wwwroot/webclient/notifications
3cxpbx/Data/Http/wwwroot/webclient/assets/manifest
3cxpbx/Data/Http/wwwroot/webclient/assets/emojione/32
3cxpbx/Data/Http/wwwroot/webclient/assets/emojione/32/_alt
3cxpbx/Data/Http/wwwroot/webclient/assets/img
3cxpbx/Data/Http/wwwroot/webclient/assets/img/picker
3cxpbx/Data/Http/wwwroot/ace-plugins
3cxpbx/Data/Http/provsubdir
3cxpbx/Data/Http/provsubdir/42/2
3cxpbx/Data/Http/provsubdir/42/2/cgi
3cxpbx/Data/Http/provsubdir/42/2/cgi/shop
3cxpbx/Data/Http/provsubdir/42/2/sht/81
3cxpbx/Data/Http/provsubdir/42/2/webuil/81
3cxpbx/Data/Http/provsubdir/sifs
3cxpbx/Data/Http/provsubdir/70/1
3cxpbx/Data/Http/provsubdir/70/1/cgi
3cxpbx/Data/Http/provsubdir/70/1/cgi/shop
3cxpbx/Data/Http/provsubdir/logo
3cxpbx/Data/Http/Templates/provider
3cxpbx/Data/Http/Templates/phones
3cxpbx/Data/Http/Templates/gateway
3cxpbx/Data/Http/Templates/fxs
3cxpbx/Data/Http/Templates/messaging
3cxpbx/Data/Http/Templates/crm
3cxpbx/Data/Logs/PbxConfigTool.log
3cxpbx/Data/Logs/unattended-upgrade.log
3cxpbx/Data/Logs/CloudServiceWatcher.log
3cxpbx/Bin/nftables.conf
3cxpbx/Bin/3CXGatewayService.config.json
3cxpbx/Bin/3CXPhoneSystem.ini
3cxpbx/Bin/nginx/conf
3cxpbx/Bin/nginx/conf/nginx.conf
3cxpbx/Bin/nginx/conf/snippets
3cxpbx/Bin/nginx/conf/snippets/50-ssl-protocols.conf
3cxpbx/Bin/Cert/Apple
3cxpbx/Bin/License
3cxpbx/Bin/3cxmediaserver.ini

 

[Mungo]

Well, I'd like to say this is a Suprise...
https://www.sentinelone.com/blog/sm...3cx-software-in-software-supply-chain-attack/
CrowdStrike

 

[Infotect-MSP]

Ping - hoping for something official from 3CX ASAP...

 

[glipschitz_summit]

At this stage, I would recommend stopping the unattended-upgrades service

systemctl stop unattended-upgrades

 

[Sean_Glasspool]

After review and reverse engineering by the CrowdStrike Intelligence Team, the signed MSI (aa124a4b4df12b34e74ee7f6c683b2ebec4ce9a8edcf9be345823b4fdcf5d868) is malicious. The MSI will drop three files, with the primary fulcrum being the compromised binary ffmpeg.dll (7986bbaee8940da11ce089383521ab420c443ab7b15ed42aed91fd31ce833896). Once active, the HTTPS beacon structure and encryption key match those observed by CrowdStrike in a March 7, 2023 campaign attributed with high confidence to DPRK-nexus threat actor LABYRINTH CHOLLIMA

 

[glipschitz_summit]

Here is a shell script to get everyone started with the cleanup.

Create a file on your server;

nano 3cx-desktopapp-cleanup.sh

# Disable 3CX Unattended-Upgrades Service

systemctl stop unattended-upgrades

# Collect the version of 3CX Desktop Apps on the Server

cd /var/lib/3cxpbx/Instance1/Data/Http/electron
ls -la * > /root/3cx-desktop-versions.log

# Remove the files

rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.dmg
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/osx/*.zip
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.msi
rm -rf /var/lib/3cxpbx/Instance1/Data/Http/electron/windows/*.nupkg

run the file

sh 3cx-desktopapp-cleanup.sh

You can check the status of the service

systemctl status unattended-upgrades

You should see it shutdown

root@3cx:# systemctl status unattended-upgrades

unattended-upgrades.service - Unattended Upgrades Shutdown

 

[VaughnMusil]

And for the Windows Endpoint Cleanup here us a Powershell script to output "True" if the detected version is installed and a Bash script to silently uninstall the software killing the process at the same time. Screenshots below are for Ninja RMM Policy.


Powershell Conditional Alert Script

$softwareName = "3CX Desktop App" # Replace with the name of the software you want to check
$version = "18.12.416.0" # Replace with the version number of the software you want to check

$installedSoftware = Get-WmiObject -Class Win32_Product | Where-Object {$_.Name -eq $softwareName -and $_.Version -eq $version}

if ($installedSoftware) {
    Write-Host "True"
} else {
    Write-Host "False"
}

Bash Removal Script

wmic product where name="3CX Desktop App" call uninstall /nointeractive

 

[PhoneEP]

3CX must at least be somewhat aware...the Desktop App download link in customer portal links over to instructions for the web client now, no longer to the installer to download.

 

[clarity2]

Has anyone seen the compromised versions get installed if you are self-hosting and auto-updates turned off so Update 7 was not installed?

We have auto-updates off on all our instances and have not installed Update 7 and we are not seeing 18.12.x versions of the 3CXDesktopApp installed or blocked by S1 anywhere.

 

[PhoneEP]

We only had one 3CX msi get blocked from S1, but it does not match the SHA1 that Crowdstrike listed as malicious. Is S1 and others blocking all 3CXDesktop App msi's then our of caution?

 

[Marco@]

Has anyone seen the compromised versions get installed if you are self-hosting and auto-updates turned off so Update 7 was not installed?

We have auto-updates off on all our instances and have not installed Update 7 and we are not seeing 18.12.x versions of the 3CXDesktopApp installed or blocked by S1 anywhere.

I'm with you on that. None of our instances, Self-Hosted & AU off, still on U6 and no impacted machines in our environments.