Solved 3CX Windows VM freezes

Discussion in '3CX Phone System - General' started by Thomas Friedl, Nov 21, 2017.

Thread Status:
Not open for further replies.
  1. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    We are running 3CX in production on a Windows Server 2016 Standard edition virtualized on a VMWare ESXi host. It's for around 40 extensions and the VM has 8GB of RAM and 4vCPUs. Performance is fine but the system itself is quite unstable with almost one freeze per week. 3CX is the only thing running on an otherwise fresh Windows Server 2016 install.

    Is there anything we can look into? Any 3CX debug logs that might be helpful?

    What happens is that the entire VM becomes unresponsive and only a hard reboot via VCenter helps. Then it works fine again for around a week until it hangs again.

    Are there any memory settings we could tweak? Or anything else we should optimize?
     
  2. StefanW

    StefanW Head of Customer Support and Training
    Staff Member 3CX Support

    Joined:
    Jun 2, 2009
    Messages:
    1,216
    Likes Received:
    88
    if you only can hard reboot the VM, then you dont need to look into 3CX logs.
    Windows Event Logs should be the stating point.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    us1 likes this.
  3. us1

    us1

    Joined:
    Oct 19, 2015
    Messages:
    80
    Likes Received:
    21
    Stefan is dead on that this is related to your underlying environment. Notepad is installed on the system but that is not the cause of the issues either.

    You did not state your build of ESXi but doing a cursory web search, I found this article on VMware's knowledgebase describing exactly what you're seeing on ESXi 6.5.0 but corrected in ESXi 6.5 U1:

    https://kb.vmware.com/s/article/2150591

    Based on the symptoms, this (or a similar issue) is probably your best bet to start troubleshooting.
     
  4. sip.bg

    sip.bg Active Member

    Joined:
    Nov 7, 2016
    Messages:
    704
    Likes Received:
    219
    I would recommend using a Debian 9 VM, it needs only 2 GB of RAM and 1 vCPU and will perform much better than Win 2016 Server. I found Win 2016 very unstable for using wth 3CX, if you insist on using Windows -- better use Win 2012 Server, 4 GB RAM and 1 vCPU will be enough.
    We are running over 50 VM under ESXi 6.0, most of 3CX instances already migrated to Debian 8 / 9. Debian 9 is better performing.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #4 sip.bg, Nov 21, 2017
    Last edited: Nov 21, 2017
    us1 likes this.
  5. us1

    us1

    Joined:
    Oct 19, 2015
    Messages:
    80
    Likes Received:
    21
    All of my instances are Debian. Been super happy with the performance and ease of management.
     
    sip.bg likes this.
  6. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    Ok thanks for the input - I found out that the issue is actually 3CX using too much memory - the VM then starts swapping and is unresponsive so I need to restart it.

    Today 3CX was using 7GB of memory and continued to eat memory. Seems a bit much for 50 extensions.

    I guess we will migrate to Debian as well.
     
  7. jimbo59

    jimbo59 Member

    Joined:
    Nov 17, 2017
    Messages:
    357
    Likes Received:
    77
    Debian is awesome. We run it on a retired desktop with a ssd drive and no issues with 50 extensions.
    Smoooth like Linux.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    We actually started with Debian but moved to Windows because of the AD Import - but actually the feature is so basic that it doesn't really pay off.
     
  9. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    Seems like the SIP Server is eating all the memory.

    3CX PhoneSystem 01 SIP Server 3 148 576 kB
     
  10. nb

    nb Support Team
    Staff Member 3CX Support

    Joined:
    Jun 7, 2007
    Messages:
    2,127
    Likes Received:
    152
    Maybe you are being attacked? This is why you get so high memory usage?

    Whoever has memory issues, we offer to check. . PM me so we login and check this out.. No, that memory usage is abnormal and it is not the case.. Something is happening. Could be a deadlock caused by a routing loop configuration or something else..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    I doubt there is an attack - it's a local server protected by our firewall. And it's been going on for weeks...

    Nevertheless - thanks for the offer. However we already migrated to Debian and it's working fine so far. The backup/restore worked really well and I was done in one hour with migrating from Windows to Linux.
     
  12. nb

    nb Support Team
    Staff Member 3CX Support

    Joined:
    Jun 7, 2007
    Messages:
    2,127
    Likes Received:
    152
    You need to be sure from where the cpu usage is coming from. Since you moved the configuration from windows tro linux and the problem went away, then I think you had a native windows issue on the OS. CPU usage are a delicate discussion. Anyway - if the problem is gone then now you are ok.

    Maybe some stuck .net update - I have seen this in the past many times.. Something installed on the server.. We dont know. If you removed the os we cannot know.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    Ok it seems the problem got worse after switching to Debian. The SIP server eats all the memory and we need to restart after a couple of hours :(

    I did check our config and I don't see any issues - but to be honest I don't know what setting could even cause an issue like this.
     
  14. Thomas Friedl

    Joined:
    Jul 14, 2017
    Messages:
    11
    Likes Received:
    0
    In case anybody else has the same problem: we figured out it was related to some phones not being provisioned correctly. Apparently they were flooding the system with subscribe (and other) messages and 3CX didn't handle it well and so the memory usage increased until it was no more. We now provisioned those phones in another way (phones in question are older aastra models) and now the problem is gone.
     
  15. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    6,016
    Likes Received:
    421
    Great to hear the issue has been resolved and thank you for sharing your solution.
     
  16. nb

    nb Support Team
    Staff Member 3CX Support

    Joined:
    Jun 7, 2007
    Messages:
    2,127
    Likes Received:
    152
    Thanks Thomas for the update. I have to jump in to correct you on some key points here because the way you report things make 3CX look like it SHOULD handle this by design. This is completely wrong.

    What you describe here is the exact same case of a person that tries to drive a car with the handbrake still in parking position and is complaining to the car manufacturer why the car is moving slowly or not moving at all..
    Overall the driver expects the car to handle this case properly!!!!

    People need to use supported phones. This company decided to keep on using old, legacy phones and admins took the decade old provisioning file for granted - dumping it into these phones as is and expecting everything to work without side-effects.

    As a result we can learn from this case how unsupported, old or legacy phones can be turned into Authenticated and Admin Approved, Denial of service Demon Bots with one .,. actually 2 scopes in life - 1) Make a call from A to B and 2) FLOOD / DOS the server with unnecessary subscribes and sip messages causing the server to eat up all the operating system's memory resources ..

    The most ironic part is here - These Subscribe messages were designed to be ONLY UNDERSTOOD (proprietary) by a discontinued Aastra pbx.. but we complain that 3CX does not handle this well. Yes of course not - they are proprietary..

    Administrators who are administering these types of phones should never dump a phone on the network and move on to the next task like we do when we work with supported phones.
    Unfortunately this is the privilege you lose when you work with unsupported or old legacy phones.

    You need erase and disable everything on the phone, put one, test it rigorously and monitor for weeks to ensure this phone behaves correctly to the standard you need. All features should be disabled and the phones should be used for basic phone calls only.

    So yes, the memory usage increased because the admin let the phones flood the server until the OS has no more memory to work with.

    This is a problem caused by wrong admin decisions. 3CX has nothing to handle here. The phones cannot be blocked because this is all lawful traffic. The admin tagged this as lawful traffic.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #16 nb, Jan 3, 2018
    Last edited: Jan 4, 2018
    us1 likes this.
Thread Status:
Not open for further replies.