Memory and CPU [DOS Attack!]

[welsh]

I'm using 3CX v9.0.13545.594 SP.2

I'm having 70-99% CPU usage, and memory usage starting around 160,000KB and increasing over the period of an hour or so up to 1,999,000KB (yes, that's 2GB)

I'm on Windows XP.
I only have 6 extensions.
My logging level is set to Low.
Everything else is fairly standard, as far as I know.

Help???

 

[leejor]

Re: Memory and CPU

Does Task Manager show that it is in fact 3CX using the CPU usage? Does the 3Cx log show anything unusual at the time? Does the same thing happen right after a PC re-boot or does it take some time to "build up"

 

[welsh]

Re: Memory and CPU

Thanks for the reply.

Task Manager shows 2GB for the 3CXPhoneService.exe process.

It happens immediately after a reboot of that PC. The 3CX process startys with High CPU usage and the memory keeps climbing.

The Trace.log file grows rapidly, and I see that there is a "login attack" possibly going on. (someone is attempting to login to a non-existant Extension with "bad credentials."

What next?

 

[leejor]

Re: Memory and CPU

What happens if you disconnect from the internet , between your modem and router? If it is an outside attack, you may need to use Wireshark to identify where they are coming from.

 

[welsh]

Re: Memory and CPU

I disconnected from the Internet (thanks for the suggestion) by way of unplugging my Firewall / Router. CPU usage dropped off quickly. I found the offending IP, and added it to my Firewall's Blacklist, and it seems like things are good again.

So... the bigger issue remains, namely that a DOS is still very possible, even iwth the new security updates that 3CX rolled out.

Any ideas for a long-term fix?

 

[leejor]

Re: Memory and CPU

Thankfully, I haven't had to look into that myself, yet. I'm sure that there are others on this forum that have had to deal with this sort of thing before, and may have some advice, besides doing what you have already done.

Are you using a static IP? If not then leave your router unplugged for a day or so and let your router pick up a new IP . Perhaps talk to your ISP and give them the IP that was making the attack, if there are similar attacks from it, maybe they will block it.

You can try using http://whois.arin.net/ui , plug in the offending IP and you should get some info on who to complain to about attacks coming from one of their customers, of course, you may not get very far when dealing with ISP's in some counties.

 

[LeonidasG_3CX]

Or you can tighten up your security settings by going to Settings > Advanced > Anti-Hacking feature and see your options there.
It would be also best to Update your PBX to the latest version as we do make fixes for Memory Leaks / DoS attacks for such scenarios.

 

[Cjay]

I had one of these attacks last night, another sip-vicious 'friendly-agent' brute force attack. The thing I found interesting was that despite 3cx blacklisting the attacking IP it continues to communicate with the attackers registration attempts (albeit refusing). I watched this on Wireshark - 10's of thousands of attempts in a very short period of time with over 1GB of data being thrown at me.

I have now firewalled every incoming port 5060 IP in my NAT router except those belonging to my VoIP providers. On doing so the sip-vicious attack stops near instantly, without some response from my end the sip-vicious scanner has no idea there is a PBX to attack. So my question is: Why does 3cx continue IP dialogue with a blacklisted IP, it only sustains the attack. Wouldn't total silence would be a better strategy?

And before you ask... In my haste to fix this attack I forgot to save the Wireshark evidence. Doh...!

20:26:55.808  Blacklisted (Too many failed auth)IP = 69.1.244.7; Failed auth: 25; unauth: 33; auth: 26
20:26:55.793  [CM102001]: Authentication failed for SipReq:  REGISTER xx.xx.xx.xx tid=-3794487932 cseq=REGISTER [email protected] / 2 from(wire); Reason: Credentials don't match, check that authorization-ID and password match the ones in extension settings

[xx.xx.xx.xx is me!]

Chris

 

[LeonidasG_3CX]

I had one of these attacks last night, another sip-vicious 'friendly-agent' brute force attack. The thing I found interesting was that despite 3cx blacklisting the attacking IP it continues to communicate with the attackers registration attempts (albeit refusing). I watched this on Wireshark - 10's of thousands of attempts in a very short period of time with over 1GB of data being thrown at me.

I have now firewalled every incoming port 5060 IP in my NAT router except those belonging to my VoIP providers. On doing so the sip-vicious attack stops near instantly, without some response from my end the sip-vicious scanner has no idea there is a PBX to attack. So my question is: Why does 3cx continue IP dialogue with a blacklisted IP, it only sustains the attack. Wouldn't total silence would be a better strategy?

And before you ask... In my haste to fix this attack I forgot to save the Wireshark evidence. Doh...!

20:26:55.808  Blacklisted (Too many failed auth)IP = 69.1.244.7; Failed auth: 25; unauth: 33; auth: 26
20:26:55.793  [CM102001]: Authentication failed for SipReq:  REGISTER xx.xx.xx.xx tid=-3794487932 cseq=REGISTER [email protected] / 2 from(wire); Reason: Credentials don't match, check that authorization-ID and password match the ones in extension settings

[xx.xx.xx.xx is me!]

Chris

We actually had a discussion regarding this internally a few days ago.
Unfortunately after that person has been blacklisted the Phonesystem HAS to reply back to the attacker / user.

I understand this puts stress on the user's network / Internet line, but 3CX MUST reply back. After the new IP has been added to the blacklist, the PBX stops sending the user Authentication Challenges no longer giving him a way to register even if he sends a Registration request with the correct Credentials and begins to send Forbidden Messages back.
- We have made sure to fix Memory Leaks / CPU issues under these kind of DoS attacks, although some fixes have been already released regarding these leaks, we do have some more that we plan to release soon with the latest Service Pack Update.

Nevertheless a good user would first take care of these kind of issues on his firewall. Less experienced users can still feel some security with the Anti-Hacking features in place.

 

[abc123]

Just to add my $0.02

The anti hacking that 3cx put in is to stop hackers getting into the system not to stop DOS attacks.

If the attack gets as far as the 3cx then the 3cx still has to look up the blacklist to see if the ip is there. This still consumes resources on the 3cx even if 3cx did not respond.

As soon as you notice these things you should block it at the firewall. A decent firewall would automatically block it for you.

We provide a lot of security analysis and consulting to small businesses and also provide a managed security appliance for a very low monthly fee. We show them how vulnerable an ip address is by opening up a new ip (on a mi fi system) and an unprotected pc and watch the wireshark (this is safe as it is our clean demo pc) and within a few minutes you will see the attacks come in on hundreds of known ports.

So many people rely on the basic firewall in their router which is woefully inadequate for a business or even home use.

We must educate businesses on how vulnerable they are because it is only a matter of time before they will be compromised.

 

[LeonidasG_3CX]

Most people also do not update to the latest Service Packs we release which provide fixes regarding DoS attacks and memory leaks.

As you understand we can only block the attacker from causing your PBX to be unusable internally, so that you can make calls to internal extensions. But 3CX is just a PBX, not a firewall, we can't do anything about packets coming in as flood from the internet taking up all your Bandwidth making your Internet / External calls unusable.

A user who has knowledge of what a PBX and an Extension is and how to configure them will surely know how to configure a firewall.
A user who can't configure a Firewall correctly has no business installing a PBX for a small-large sized business. There are IT professionals you can hire for that in my opinion.
There's a difference between users who know how to configure a firewall but don't bother, and users who have no idea what a firewall does.

 

[sigma1]

Just to add my $0.02

As soon as you notice these things you should block it at the firewall. A decent firewall would automatically block it for you.

We provide a lot of security analysis and consulting to small businesses and also provide a managed security appliance for a very low monthly fee. We show them how vulnerable an ip address is by opening up a new ip (on a mi fi system) and an unprotected pc and watch the wireshark (this is safe as it is our clean demo pc) and within a few minutes you will see the attacks come in on hundreds of known ports.

So many people rely on the basic firewall in their router which is woefully inadequate for a business or even home use.

We must educate businesses on how vulnerable they are because it is only a matter of time before they will be compromised.

I am not aware of any intelligent router that comes up with filters all by itself. If there is one please let me know the make/model.

Routers follow filter rules, the basic rules are based on originating IP address, destination IP address, originating port, destination port and protocol.

This issue has been raised several times and the most simple solution is to restrict all traffic with port 5060 to the range belonging to your VoIP provider. Open port 5090 (Tunnel) in the event that you have remote users, or if your router/firewall supports VPN, take advantage of it. You can keep it simple by using PPTP as the most basic VPN (easy setup).

The basic rule of keeping things simple still applies, only port fwd the ports you need (for the most part apps running on port 80, 443, 25,110,143 are secure, or at least use a web/email server that is reputable). Setup a conditional 5060 FWD. Don't waste money on overly fancy routers... They will likely cause other issues (Sonicwall comes to mind ;-).

 

[semn]

A very good firewall/vpn-gateway is Cisco Asa 55xx. This boxes can detect and block, fully automatically DoS attacks.For small business i recommend ASA5505 and for bigger business i recommend ASA5510.