Solved 3CX Windows VM freezes

Status
Not open for further replies.

Thomas Friedl

Joined
Jul 14, 2017
Messages
11
Reaction score
0
We are running 3CX in production on a Windows Server 2016 Standard edition virtualized on a VMWare ESXi host. It's for around 40 extensions and the VM has 8GB of RAM and 4vCPUs. Performance is fine but the system itself is quite unstable with almost one freeze per week. 3CX is the only thing running on an otherwise fresh Windows Server 2016 install.

Is there anything we can look into? Any 3CX debug logs that might be helpful?

What happens is that the entire VM becomes unresponsive and only a hard reboot via VCenter helps. Then it works fine again for around a week until it hangs again.

Are there any memory settings we could tweak? Or anything else we should optimize?
 
if you only can hard reboot the VM, then you dont need to look into 3CX logs.
Windows Event Logs should be the stating point.
 
  • Like
Reactions: us1
We are running 3CX in production on a Windows Server 2016 Standard edition virtualized on a VMWare ESXi host. It's for around 40 extensions and the VM has 8GB of RAM and 4vCPUs. Performance is fine but the system itself is quite unstable with almost one freeze per week. 3CX is the only thing running on an otherwise fresh Windows Server 2016 install.

Is there anything we can look into? Any 3CX debug logs that might be helpful?

What happens is that the entire VM becomes unresponsive and only a hard reboot via VCenter helps. Then it works fine again for around a week until it hangs again.

Are there any memory settings we could tweak? Or anything else we should optimize?

Stefan is dead on that this is related to your underlying environment. Notepad is installed on the system but that is not the cause of the issues either.

You did not state your build of ESXi but doing a cursory web search, I found this article on VMware's knowledgebase describing exactly what you're seeing on ESXi 6.5.0 but corrected in ESXi 6.5 U1:

https://kb.vmware.com/s/article/2150591

Based on the symptoms, this (or a similar issue) is probably your best bet to start troubleshooting.
 
I would recommend using a Debian 9 VM, it needs only 2 GB of RAM and 1 vCPU and will perform much better than Win 2016 Server. I found Win 2016 very unstable for using wth 3CX, if you insist on using Windows -- better use Win 2012 Server, 4 GB RAM and 1 vCPU will be enough.
We are running over 50 VM under ESXi 6.0, most of 3CX instances already migrated to Debian 8 / 9. Debian 9 is better performing.
 
Last edited:
  • Like
Reactions: us1
I would recommend using a Debian 9 VM, it needs only 2 GB of RAM and 1 vCPU and will perform much better than Win 2016 Server. I found Win 2016 very unstable for using wth 3CX, if you insist on using Windows -- better user Win 2012 Server, 4 GB RAM and 1 vCPU will be enough.
We are running over 50 VM under ESXi 6.0, most of 3CX instances already migrated to Debian 8 / 9. Debian 9 is better performing.
All of my instances are Debian. Been super happy with the performance and ease of management.
 
  • Like
Reactions: sip.bg
Ok thanks for the input - I found out that the issue is actually 3CX using too much memory - the VM then starts swapping and is unresponsive so I need to restart it.

Today 3CX was using 7GB of memory and continued to eat memory. Seems a bit much for 50 extensions.

I guess we will migrate to Debian as well.
 
Debian is awesome. We run it on a retired desktop with a ssd drive and no issues with 50 extensions.
Smoooth like Linux.
 
We actually started with Debian but moved to Windows because of the AD Import - but actually the feature is so basic that it doesn't really pay off.
 
Seems like the SIP Server is eating all the memory.

3CX PhoneSystem 01 SIP Server 3 148 576 kB
 
Maybe you are being attacked? This is why you get so high memory usage?

Whoever has memory issues, we offer to check. . PM me so we login and check this out.. No, that memory usage is abnormal and it is not the case.. Something is happening. Could be a deadlock caused by a routing loop configuration or something else..
 
I doubt there is an attack - it's a local server protected by our firewall. And it's been going on for weeks...

Nevertheless - thanks for the offer. However we already migrated to Debian and it's working fine so far. The backup/restore worked really well and I was done in one hour with migrating from Windows to Linux.
 
You need to be sure from where the cpu usage is coming from. Since you moved the configuration from windows tro linux and the problem went away, then I think you had a native windows issue on the OS. CPU usage are a delicate discussion. Anyway - if the problem is gone then now you are ok.

Maybe some stuck .net update - I have seen this in the past many times.. Something installed on the server.. We dont know. If you removed the os we cannot know.
 
Ok it seems the problem got worse after switching to Debian. The SIP server eats all the memory and we need to restart after a couple of hours :(

I did check our config and I don't see any issues - but to be honest I don't know what setting could even cause an issue like this.
 
In case anybody else has the same problem: we figured out it was related to some phones not being provisioned correctly. Apparently they were flooding the system with subscribe (and other) messages and 3CX didn't handle it well and so the memory usage increased until it was no more. We now provisioned those phones in another way (phones in question are older aastra models) and now the problem is gone.
 
Great to hear the issue has been resolved and thank you for sharing your solution.
 
In case anybody else has the same problem: we figured out it was related to some phones not being provisioned correctly. Apparently they were flooding the system with subscribe (and other) messages and 3CX didn't handle it well and so the memory usage increased until it was no more. We now provisioned those phones in another way (phones in question are older aastra models) and now the problem is gone.

Thanks Thomas for the update. I have to jump in to correct you on some key points here because the way you report things make 3CX look like it SHOULD handle this by design. This is completely wrong.

What you describe here is the exact same case of a person that tries to drive a car with the handbrake still in parking position and is complaining to the car manufacturer why the car is moving slowly or not moving at all..
Overall the driver expects the car to handle this case properly!!!!

People need to use supported phones. This company decided to keep on using old, legacy phones and admins took the decade old provisioning file for granted - dumping it into these phones as is and expecting everything to work without side-effects.

As a result we can learn from this case how unsupported, old or legacy phones can be turned into Authenticated and Admin Approved, Denial of service Demon Bots with one .,. actually 2 scopes in life - 1) Make a call from A to B and 2) FLOOD / DOS the server with unnecessary subscribes and sip messages causing the server to eat up all the operating system's memory resources ..

The most ironic part is here - These Subscribe messages were designed to be ONLY UNDERSTOOD (proprietary) by a discontinued Aastra pbx.. but we complain that 3CX does not handle this well. Yes of course not - they are proprietary..

Administrators who are administering these types of phones should never dump a phone on the network and move on to the next task like we do when we work with supported phones.
Unfortunately this is the privilege you lose when you work with unsupported or old legacy phones.

You need erase and disable everything on the phone, put one, test it rigorously and monitor for weeks to ensure this phone behaves correctly to the standard you need. All features should be disabled and the phones should be used for basic phone calls only.

So yes, the memory usage increased because the admin let the phones flood the server until the OS has no more memory to work with.

This is a problem caused by wrong admin decisions. 3CX has nothing to handle here. The phones cannot be blocked because this is all lawful traffic. The admin tagged this as lawful traffic.
 
Last edited:
  • Like
Reactions: bravad and us1
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,880
Messages
589,533
Members
164,735
Latest member
998742