SIP Inbound Stops Working After A Week Or Two

Discussion in '3CX Phone System - General' started by JCLloyd, Dec 21, 2017.

Tags:
Thread Status:
Not open for further replies.
  1. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Our 3CX system can't seem to much more than a week before the system just stops accepting inbound traffic from our SIP provider. I found that I can bounce the "3CXPhoneSystem##" service with associated "3CXQueueManager##" and "3CXIVR##" services preceding it, and the inbound trunks again take calls. This last time, the IVR did not play audible messages... just blank for the length of the messages.

    Right now, I am trying a weekend reboot with a delayed service bounce script of the afore-mentioned services shortly after. But, this just doesn't seem right. There has to be a more simple solution, and I could use some thoughts... I am not sure what else would be causing the resource issues.

    I am currently porting my anti-virus exclusions to Windows Defender from Kaspersky, verifying the firewall settings, and removing Kaspersky.

    Just in case... Hardware:
    - Core i3-6200
    - 8GB RAM
    - 256GB SSD
    - Windows 10 Pro, on a Domain
    - Dedicated to PBX

    Thank you, in advance, for any advice...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. decibel

    Joined:
    Aug 26, 2015
    Messages:
    9
    Likes Received:
    2
    Who is your ISP and SIP provider? I wonder if we are having the same problem.
     
    JCLloyd likes this.
  3. jvivona

    Joined:
    May 18, 2014
    Messages:
    22
    Likes Received:
    7
    I had something like this, but I was running as a guest in Hyper-V and my time was out of sync - so the packets were all over the place and we couldn't get any calls & even when we called each other is wasn't working. I'd reboot and then it would work.

    Have you checked your time / timezone? Are you sure its accurate?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    JCLloyd likes this.
  4. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Broadvoice, Los Angeles County
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Good call... As of right now, I am off about a quarter of a second. I just restarted about 22 hours ago. I'll check again to see if it drifts with days... which would be bad.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. Nick Galea

    Nick Galea Site Admin

    Joined:
    Jun 6, 2006
    Messages:
    1,889
    Likes Received:
    190
    You should never run Anti Virus on a server system and certainly not on a PBX. This might not be the problem here - not enough data is available - but disabling anti virus will need to be done anyway
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Nick, I understand your concern regarding anti-virus applications... (**)

    Back to the issue of my system forgetting how to answer an incoming SIP line... I would think that something else is happening, and consuming or periodically redirecting resources that 3CX needs. I need a 3CX coder-type who has a grasp on the processes behind the Inbound Rules, what they rely on, and what may be happening to cause an inbound rule to lose the ability to interact with a SIP trunk. It may be as simple as my Saturday project of removing Kaspersky, and configuring Windows Defender to not get in the way. But, I am just going on a hunch.

    ** -
    . I am from the camp where firewalls and anti-virus/anti-malware applications are an unfortunate necessity of life, and have spent the last decade mastering settings to allow them to exist in my environments. But, I am increasingly of the camp realizing that anti-virus applications are effectively a "man-in-the middle" of all data traversing the operating system. They are a double-edged sword that tries to protect with the potential of causing grief and harm, if not properly selected or configured.
    . Regardless, it is another program running on the system. I prefer the minimum overhead approach, when possible. Removing Kaspersky will allow Windows 10 v1709 to fully engage the embedded Defender suite, which includes its own anti-virus/anti-malware engine. It is also present in Windows Server 2016. Given those choices, I would prefer the Microsoft solution for its own operating system, as opposed to a third party solution... along with end-user training.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    I am back... Happy New Year!

    I am still grappling with the SIP issue. I can only run about 9 to 12 days before the 3CX stops responding to calls. All services appear to be running, but if I bounce "3CXQueueManager01", and answer "yes" to also bouncing "3CXIVR01" and "3CXPhoneSystem01"... inbound calls again ring in through the inbound rules.

    Tried a weekly restart of the system and running a script that bounces the three services with "3CXQueueManager01" last to stop and start. The "IVR Server" service had to be manually started, which I did through the 3CX services panel on the dashboard. Not quiet there...

    Is anyone else having (or have had) this issue? Any ideas? This is my last hurdle before approving 3CX for prime time and putting into production.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. billtodd77

    Joined:
    Jun 23, 2017
    Messages:
    3
    Likes Received:
    0
    I had this issue a while back on v14. Turned out to be a NAT traversal issue. Does the firewall checker pass all the way through?
     
  10. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    The firewall test has passed several times, with no fails (Green). Unless there is something intermittent, I am not seeing anything. Good call, though...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. flex2

    Joined:
    Dec 12, 2014
    Messages:
    4
    Likes Received:
    1
    I am also having this issue. This is a directly attached SIP trunk via a copper hand off, not of the hosted (read: through the firewall). Spectrum is the provider. So I know its not NAT or FW.
    I get approx 12-15 days before the audio goes belly up. I can still see calls coming into the system but nobody can hear the caller / callee. The system just stops passing audio. I have roughly 150 3CX PBX's in product to date, no others with this issue. We pretty much cookie cutter our deploy prep work and hardware, however there is a vairance with the hardware of this machine 3cx runs on BUT it has a brother at another non related company (built at the same time), with no issues.

    3CX support to date has had no suggestions other than to collect PCAP's and Verbose log files while the issue is happening. Unfortunately something I am doing in the PCAP captures is wrong and they dont see any data in them which I find odd. So that has been ongoing for months now. I do have 2 interfaces going, 1 attached to the LAN which the phones are on, and 1 connected to the carrier's SIP gateway.

    I did learn something new this last time around, the end users reported that the audio gets choppy hours before the black out happens. Check with your EU's to see if this is similar behavior on your side.

    I have updated NIC drivers, windows is fully patched, no A/V (doesn't belong on servers that are headless anyway), I've set all my power settings and removed windows firewall <- all of that is standard for our deployments.

    This past week the EU's were reporting that calls were choppy all the time. So something got worse somehow. Again, our SIP does NOT ride the open internet. Its a dedicated circuit with no WAN access of any sort, goes from client to node to service provider on a closed network.

    So this last weekend I simply just decided to go off on my own with this issue and I reloaded just 3CX on this system. It seems to have corrected the choppy audio issues that were now present daily instead of just hours before the black out. Now I sit and wait to see if anything turns up +/- 15 days from now.

    Another note, this system has had issues since v15sp1. We had a problem with the mediaserver.exe service, 3cx came into the pbx and replaced the exe. Didnt help. I suspect the issue still is to do with the mediaservice.exe service.
     
    JCLloyd likes this.
  12. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Flex2, I'm sorry that you have a similar issue, but I'm glad I am not alone. Enough of us with this issue may pinpoint the actual source of this issue. Together, we will warrant a 3CX correction.

    From what I am experiencing, this appears to be a application and OS related issue. I don't see how collecting traffic samples will point to the cause of the issue. Traffic is there until 3CX stops managing it.

    Since my issue seems to be centered around the IVR Server, I tried putting in a delayed startup on the IVR Server service and its two dependents. Last night's scheduled system restart was successful, but I will need to repeat that tonight and tomorrow night to make sure this is a viable 'reset' before the time period the system becomes unusable.

    I'll keep a watch on the services, more closely. I don't think that I have had issues with the media server, but maybe I am... Maybe, there are more services being affected than the one drawing my attention.
    . I think our issue actually began around mid-November, right after upgrading Windows 10 to v1709.
    That might have been around the time I upgraded to SP1. I had waited, and SP2 came out really soon after that, for me. It seems there was an update against SP2 within a week after I installed it. I started out with v15.5-SP0, back in August. Windows 10 was on v1703 and Kapsersky AV. I don't remember these issues, but was in a heavy learn mode. Difficult to say if I was experiencing this issue at that time.
    The Defender Security Center is notably upgraded to fit into their cloud 'Defender ATP' service. This will be the same for Server 2016. I am not sure that just disabling antivirus will get it out of the way of 3CX. Here are some things I am looking at, right now, and changing to see how it affects 3CX. This might be an area 3CX should look at:
    - App & browser control;
    - [Off; default is On] Check apps and files {Might affect SIP traffic...?}
    - Exploit Protection {Under App & browser control}
    - [Off; Default is On] Control flow guard (CFG) - Requires a system restart...
    ************
    ... 3CX... What about the other settings? Are any of these potentially interfering with 3CX functionality?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Flex2, I have 4 reboots that have brought the system successfully back up from a scheduled restart. The "Delayed Start" for the services pertaining to my issues has resolved my needing to intervene. That doesn't resolve the initial issue we both are experiencing where audio stops processing for you and inbound rules stop picking up calls for me. I am hoping that the weekly weekend reboot is soon enough to not break the call flow during production hours.

    Just when I thought Windows patches may be causing an issue with 3CX, OS updates to address "Meltdown" and "Spectre" come out. I'll be installing that in the next couple of hours... Let's see if that changes anything.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    3CX...? Any ideas, here? I am getting some management concern about system stability and reliability. I wanted to run this out a few weeks to see if I have everything I corrected, but I m being asked to come up with a quick solution to something that takes two or 3 weeks to determine is ready to put into production.

    What about the Linux distribution? If I get off of Windows 10 and go with the Debian Distro image...
    -1- Is it likely to NOT have these unstable service issues?
    -2- Can I restore my Windows 10 backup of 3CX to the Debian Linux version and get everything back?

    Maybe, I should start another thread about switching OS versions...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  15. us1

    us1

    Joined:
    Oct 19, 2015
    Messages:
    80
    Likes Received:
    21
    All of my installs are Linux and I have been very happy with their stability. Many others on the forums have a significant install base of Linux machines and seem to be similarly happy as well. I do have an open post out here about once in a while a system will just a busy signal that requires some services to restart while rebooting does not actually solve the problem. It certainly is not a showstopper as it has been months since this happened for this particular customer.

    You should be able to backup the configuration, install the Linux version, and restore using your backup from the Windows install. It certainly won't hurt anything to try it out but I suspect the issue will go away for you.
     
    JCLloyd likes this.
  16. sip.bg

    sip.bg Active Member

    Joined:
    Nov 7, 2016
    Messages:
    704
    Likes Received:
    219
    Quick answers to your questions:
    1. Debian is much more stable and preferable for me than desktop OS like Windows 10. I also prefer Debian than Windows Server OS, especially Windows Server 2016. Install Debian 9 from 3CX ISO.
    2. Yes you can restore with no issues. Keep in mind that changing FQDN and certificate needs first license key to be released from FQDN. Nothing necessary to do if you restore from your backup including license key and FQDN.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Edi Kojsi and JCLloyd like this.
  17. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Thank you both, us1 & sip.bg... That sounds like what I was hoping to hear. I do nightly full backups, but will do a manual full backup, before taking the system down to rebuild. That should have everything I need, short of setting the fixed IP address. I do have a spare SSD to install to and keep the Windows 10 environment, just in case I do something bone-headed or need to fall back to it for some reason.

    It would be nice to let this thread hang for a while and see if others chime in on whether Windows 10 is giving them issues that Debian is not.

     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    I'm Linux bound! The change over was amazingly painless, compared to the expectation of a Windows install/upgrade... Thank you to us1 and sip.bg for not listing any hesitations in the choice.

    My first impression is the compact non-gui install is resource light allowing the hardware for much more head room than Windows afforded my instance. Some for those curious:
    - Linux still needs updates,and this URL will help - https://www.3cx.com/blog/voip-howto/os-security-updates-debian/
    - This is a command line install. For us older geeks used to DOS & CMD, this is familiar territory. New commands, but same motions. For newer IT persons getting used to Microsoft's Power Shell, this is nowhere near as complex!
    - Backups: It seemed like a lot of extra effort to command line format and mount a second drive for backups. Unlike Windows, the Debian Distro is light enough to warrant a re-install with an import of the latest backup. So, I set up a Google Drive and am sending my backups (encrypted) up there. I dinked around with my Synology NAS FTP settings, but this was much easier... and the 3CX recommendation. Google Drive security should be fine for this.

    Let's see if I make it past 2 weeks without an issue. I don't want to jinx this by prematurely announcing success...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    us1 likes this.
  19. us1

    us1

    Joined:
    Oct 19, 2015
    Messages:
    80
    Likes Received:
    21
    Glad to hear it is working out well for you thus far. If you use LetsEncrypt and need assistance setting it up and automating the renewal, let me know.
     
  20. JCLloyd

    JCLloyd New Member

    Joined:
    Oct 5, 2017
    Messages:
    112
    Likes Received:
    19
    Day 15 and the system is still rocking! This is longer than I was getting with Windows 10 after upgrading to v1709 and taking 3CX v15.5 from SP1 to SP2.

    The system has been running about 48 hours in production mode, and seems to be using about half the RAM and CPU that Windows did in testing. In about a week, I'll look into getting SNMP set up, so that I can get system statistics logged.

    Anyone thinking about making the Linux move and waiting for someone to give a positive experience, here it is. Customization is by command line. I have found just about everything I've needed to set things up by searching the Internet.

    This announcement is a little premature, as I wanted to wait until February 5th to announce. I was worried that the thread would close if I didn't update, soon.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    NickD_3CX likes this.
Thread Status:
Not open for further replies.