3CX compromised and toll fraud

[IT247]

We have some 3cx instances hosted with a 3rd party. Three times in the past six weeks are so, we have had extensions compromised and fraudulent toll calls were able to be placed until the sip provider noticed this and shut it down. (happened after hours and they caught it in about 4 hours). High tolls were charged.

I'm trying to determine how this is happening and harden the system.

Originally we were using default 3 digit extension numbers as auth id with 5 character random alpha numeric characters for pw. Also the default 3cx security of lockout for 30 minutes after 25 failed attempts was in place.

After the first compromise, this was changed to use 16 alpha numeric mixed case strings (for Auth ID and Password) and lockout was configured to lockout for a year after 5 failed attempts. The other passwords for Yealink web ui and others were changed also.

Since that last change at least one of the same extensions that was compromised previously was compromised again. Very doubtful if not impossible to brute force random 16 character STRONG user id and separate 16 character STRONG pw in five attempts.

The 3rd party host is pointing to security issues w/ 3cx. Specifically they are suggesting the provisioning url is public facing and this is being brute forced. I reviewed the provisioning url they provided and it contains two separate random strings in the url, so I think brute force is highly unlikely. However, if the url is truncated to include just the first random string and then the macaddressofphone.cfg is appended to it, the full config is visible. Grant it a bad guy would need to figure out the first random string and also know the mac address of a phone on that system which doesn't seem likely unless this information was leaked somewhere.

Even with the random strings in the provisioning url, should this be public facing at all? I asked about locking it down to an ip address, but was told that would break connections for the mobile and desktop apps.

If it was leaked and now that information is secure, how would one change the provisioning url so bad actors in possession of it can't use that compromised link anymore?

Any help or thoughts are appreciated.

 

[Marari]

@IT247

As for locking it down to an IP address, the mobile softphone client uses port 5090 (by default) as a tunnel port to the PBX. If you leave 5090 open, you should be fine and lock down port 5060/5061 to just your voip provider(s) IP address(es).

Is your PBX cloud hosted with a provider? Do you have access to the admin management console? Who else has access to the management console?

Are you using 3CX SMTP or your own SMTP for the welcome email?

When you re-register an extension, a welcome email is usually sent out that contains the connection specific info. Is it possible that someone's mail credentials have been compromised?

There are a lot of possible ways for someone to 'backdoor' in to a system - all of them start at the user and admin levels.

 

[DSXDATA]

A common compromise happens because a user's email is hacked and then data-mined for the welcome message which has the extension credentials in the clear. The credentials are sold on dark web so there is a usually a several day/week time spam between email hack and the intrusion.

 

[IT247]

Thanks for your reply,

The PBX is cloud hosted with a provider. I have access to the admin management console and it has a strong password. Besides me, it would be the provider with access. During all incidents only a subset of the extensions were showing evidence of compromise not the entire install base so I'm guessing this didn't come from management console login compromise.

As I understand it, the extensions register over 5060 to the cloud 3cx instance. If we were to lock it down to just the voip provider (sip service?), how would these extensions register with 3cx.

SMTP was with cloud provider, but that was changed to use our SMTP with authentication and strong password and only from their network. After the previous incidents, welcome emails were sent to many other extensions and those aren't showing compromise.

One of the extensions in question has 2FA for email and there has been no evidence the mailbox was compromised, so I don't think welcome email in recipient's mailbox is source.

 

[Marari]

As I understand it, the extensions register over 5060 to the cloud 3cx instance. If we were to lock it down to just the voip provider (sip service?), how would these extensions register with 3cx.

You use a Session Border Controller which communicates with the cloud-based PBX over the tunnel port (5090 by default). The nice thing about the SBC is that you don't have to poke any holes in the firewall/router of the phone network - the request is outbound and NAT points it back to the SBC.

As for the other issues, you have a hole somewhere. I suspect one of the following possible scenarios:

- A user was phished and gave up their credentials for their email account.
- A user inadvertently allowed a trojan or keylogger to be installed on their system.
- The welcome mail was forwarded or sent over unencrypted channels (not using SSL/TLS for email)
- There is a malicious user or purposeful tampering by someone with access.

There could be other ways too. I'm simply suggesting that, in all of my experience where a breach has occurred, a user has been at fault.

 

[florink]

It is almost impossible for anyone to just brute-force multiple provisioning URLs (even one). From what you are describing it seems like the attacker has already what they need to access those extensions. If they got the provisioning URLs, it wasn't through the PBX.

Also logs is your best friend. Don't just play the guessing game. The logs will either give you a definite answer as to what happened, or guide you to a certain direction. You might also enable extra logging to see what requests are made to the server.

You might also consider changing the host of those instances as you cannot trust or know who has access to the machine..

 

[IT247]

I had verbose logging enabled prior to this last incident. I see the ip (from Palestine) that registered and began making the fraudulent calls. I see a bunch of others that got blacklisted after 5 attempts. The previous compromises were from the same region, so I'm thinking this is the same group. I'm not sure what else I should be checking in the logs.

From the previous breach, I do believe the attacker does have access information for the extensions in question (provisioning url and mac address). With the provisioning url and mac address of a device, I believe they were able to again pull down a config and obtain the new hardened plain text passwords present within.

From what I understand, there is no way to change the provisioning url to some other random string other than reinstalling to a new instance and setting everything up from scratch. I"m told a backup and restore to new server would retain the compromised provisioning url. Can anyone confirm this?

I would look into deploying an sbc on premise, but the current host is telling me you can't lock down 5060 inside of 3cx to specific ips, and there hosting infrastructure doesn't have that capability currently.

 

[cobaltit]

You don't need to lock down 5060 to specific IPs. You simply check the box 'Disallow use outside of LAN' for the extension. Tunnel connections are considered internal.

 

[cobaltit]

Another option is to move the instances away from the provider to something you have a little more control over. Since you are already in the cloud everything should be provisioned via FQDN so it shouldn't be too difficult. AWS (Not lightsail)/Azure/GCE would give you more granular firewall control if you did run into a scenario where you wanted to block IPs

 

[IT247]

Thanks for the replies,

If I select disallow use outside of the lan, how would my extensions on premise and on a different lan subnet connect to the cloud 3cx instance on its own subnet?

 

[cobaltit]

So the disallow outside of LAN is per extension. So you would set that for all your softphone/3CX Client extensions and those that are behind a SBC. If you can setup a routed VPN using RFC1918 IP ranges those will also connect with that enabled. Everything else will have to have that box unchecked. Might not work for all phones but will greatly reduce the number of endpoints that can be compromised. You can also look at disabling calls after hours for extensions.

 

[IT247]

@cob

So the disallow outside of LAN is per extension. So you would set that for all your softphone/3CX Client extensions and those that are behind a SBC. If you can setup a routed VPN using RFC1918 IP ranges those will also connect with that enabled. Everything else will have to have that box unchecked. Might not work for all phones but will greatly reduce the number of endpoints that can be compromised. You can also look at disabling calls after hours for extensions.

Thanks for your advice. The Hosted PBX provider has an SBC on their end, but if I'm understanding your suggestion for disallowing extension outside of the Lan, I would need to deploy and SBC on premise as well? Is that correct? If so, I would probably want to find a gadget that would do this and provide wan failover when primary isp goes down.

 

[cobaltit]

@IT247

Yes you would want to deploy a 3CX SBC which is a little different than a standard SBC. A 3CX SBC is not an edge device so you wouldn't worry the failover, etc. That would be handled by your firewall/router. Rather the 3CX SBC uses the 3CX tunnel protocol to communicate with 3CX thus any endpoints behind it are considered internal.

https://www.3cx.com/docs/3cx-tunnel-session-border-controller/

You can deploy it on a Raspberry Pi (current version) which is good for around 20 extensions depending on BLF usage. Beyond that you can install it on hardware or in a VM which should be good for up to 50 extensions (again depending on BLF usage). If you have more endpoints than that then you would need to use multiple SBCs.

 

[leejor]

I would need to deploy and SBC on premise as well? Is that correct?

Yes. Depending on the number of extensions, it could be done very economically with a Raspberry Pi.

https://www.3cx.com/docs/3cx-tunnel-session-border-controller/

 

[Lee Cramman]

I would be thinking very hard about finding a new VPS provider that offers granular firewall control - (I think it's unforgivable not having this feature). Your current provider appears to be trying to blame 3CX for it's own shortcomings.

There is quite a bit that can be done to harden an install of 3CX starting with the use of an SBC but without a decent firewall you will struggle.

edit: I see that in V16 3CX will require authentication in order to send the user agent.

 

[dandenson]

SBC SBC SBC. This is the best option. Disallow remote and use the SBC. Phones behind the SBC register and the mobile apps use a tunnel so they register.

 

[cobaltit]

edit: I see that in V16 3CX will require authentication in order to send the user agent.

Curious how you thought this was going to help the OP's situation? Hiding the UA until after authentication isn't going to change the fact that the attacker seems to already have the extension credentials. It's basically security by obscurity which really isn't much security at all.

In the OPs case, assuming they already already know the provisioning URL and the MAC address no amount of resetting credentials is going to fix things. One can only change the provisioning URL which may or may not work depending on how the attacker found the URL in the first place or utilize the manufacturers method of encrypting the config files which 3CX doesn't support at the moment.

 

[jasit]

A couple things we did that might help,

1. under security Settings, Change the allowed country codes, uncheck the ones that you don't want.
2. In the outbound rules add all the 900 numbers and toll numbers you want to block, make sure they are at the top of the list
3. edit the blacklist time and change it to 8640000 instead of 30 minutes.
4. Make sure you are emailed when a blacklist occurs.
5. If you see a blacklist come from the same ip range, go edit the black list and block the entire range.

Jasit

 

[Lee Cramman]

Curious how you thought this was going to help the OP's situation? Hiding the UA until after authentication isn't going to change the fact that the attacker seems to already have the extension credentials. It's basically security by obscurity which really isn't much security at all.

In the OPs case, assuming they already already know the provisioning URL and the MAC address no amount of resetting credentials is going to fix things. One can only change the provisioning URL which may or may not work depending on how the attacker found the URL in the first place or utilize the manufacturers method of encrypting the config files which 3CX doesn't support at the moment.

It doesn't help, it was just supplemental information that I discovered yesterday that I thought the OP might find interesting.

Frankly, the OP needs an SBC and a decent firewall / hardening of the PBX. Running without at least one or, preferably, both is a recipe for trouble. I've never come across a VPS provider that doesn't provide a decent firewall and wonder if this isn't just a miscommunication between parties or a first line support person (from the VPS provider) that doesn't know their stuff.

I would also be thinking about how the rogue party might have obtained the credentials in the first place because I don't believe it was by brute force or chance and that route must be closed down.

Just a thought - on a few occasions I've bought 'reconditioned' SIP phones that were properly factory reset but re-provisioned themselves over STUN to give me access to the original owners PBX when connected to the internet because someone didn't delete the phones from the PBX when they were removed. I've even dialled someone senior sounding from the phone book and explained how and why I have access to their PBX. It's an interesting conversation to have but if I were a little less moral I could exploit it to my own ends... By any chance has your organisation disposed of any phones recently?