Email notification if SBC stops communicating

Discussion in 'Ideas' started by mcbsys, Feb 25, 2017.

Email notification if SBC stops communicating 5 5 26votes
5/5, 26 votes

  1. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    I recently encountered an issue where the SBC encountered an error and stopped communicating (details in this post). In spite of errors visible in both the SBC log and the 3CX server tunnel log, I only discovered the issue when the I tried to make a call and the call failed.

    Please monitor the SBC client and server for successful communication and optionally send an email alert when the connection fails:
    • On the server side, this should be configurable under Settings > Email Settings > Notifications.
    • On the client side, the email settings would probably need to be configured by the installer and stored in 3cxsbc.conf.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    DolbySR and uptime1 like this.
  2. Jazz Oberoi

    Joined:
    Feb 17, 2017
    Messages:
    12
    Likes Received:
    5
    +1 Have had this happen to me as well. Only found out when the users started to complain.
     
  3. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    Re-reading that thread, it seems that the OP is assuming that if the SBC fails, phone registrations fails. In my case, that did not happen--phones looked registered, but calls failed. So whatever is done to monitor the SBC, it needs to confirm actual bi-directional communication. Listing each connected SBC as if it is a trunk or extension sounds good as long as they can also alert if down. The info IS in the logs, i.e. 3CX could identify the failure on both the SBC and the server side; it just needs to be surfaced and alerted appropriately.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Nathan Boyd

    Joined:
    Feb 2, 2017
    Messages:
    34
    Likes Received:
    8
    Eventually the extensions will lose their registrations as their timers run out and they go to re-register. So you are correct, there is a time period where the Tunnel can be disconnected and the Phone still think they are registered. This is why integrated SBC monitoring is important, currently the Extension Registration notification is the only automated method, and that only after a time lag.
     
  5. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    I'm pretty sure this was down for hours but still showing registered. Maybe SIP was passing through but not RTP? It was a very strange SSL error that was solved by rebooting the server (not the SBC). Details: 3CX SBC Invalid signature received: 0x10317.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #6 mcbsys, Mar 3, 2017
    Last edited: Jun 27, 2017
  6. Lev

    Lev New Member

    Joined:
    Sep 19, 2015
    Messages:
    116
    Likes Received:
    14
    +1
    Proper Monitoring of the internal infastructure of the whole 3cx system is mandatory and should not be left behind.
     
  7. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    I am again unable to get the SBC to connect to the server due to an SSL error. This is with server 15.5. I was on SBC 15.0 but upgraded to SBC 15.5 and see the same thing. And still no notification from the server, although it is obvious from the 3CXTunnel log that the connection has been failing all day with messages like this:

    Code:
    16:26:19.793|00000578|Error|TLSTransp.cpp(360): SSL_do_handshake(680@TLS(HS)) has returned -1, error code 1
    16:26:19.793|00000578|Error|TLSTransp.cpp(362):  * SSL error: error:00000001:lib(0):func(0):reason(1)
    16:26:19.793|00000578|Error|TLSTransp.cpp(362):  * SSL error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
    16:26:19.793|00000578|Error|ConnMgr.cpp(1154): IncomingTCP: Connection 1328<-<remote site IP>:59758:680 read failure
    16:26:19.793|00000578| Info|ConnMgr.cpp(1167): IncomingTCP: Removing 1328<-<remote site IP>:59758:680 . reason - processing failed (closed or failed connection)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Nathan Boyd

    Joined:
    Feb 2, 2017
    Messages:
    34
    Likes Received:
    8
  9. OCWI

    OCWI New Member

    Joined:
    Jan 17, 2017
    Messages:
    159
    Likes Received:
    46
    This isn't a feature its a necessity. Multi million dollar companies rely on SBCs every hour of every day. By the time they notify us via our own ticketing system its WAY to late to not have an answer.
     
  10. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    Wow this is huge. It seems that the SBC issues run deeper than notifications. I can see what I assume is the 170625 expiration date of the client certificate in the 3cxsbc.exe version 15.5.1136.6 file. Time to switch to external PKI certs, not secret and unmanageable embedded certs?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. OCWI

    OCWI New Member

    Joined:
    Jan 17, 2017
    Messages:
    159
    Likes Received:
    46
    It is huge indeed.

    3cx Maintenance license is supposed to cover these certs. Maintenance is not cheap.

    Either way, this "feature" is a must for anyone installing these for larger businesses.
     
  12. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    The SBC certificates are different from the customer-specific certs. There is a certificate embedded within the 3cxsbc.exe file and probably another on the server side, which means it's the same certificate pair for everyone worldwide. Apparently it expired 6/25/2017.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. teichhei

    Joined:
    Oct 11, 2009
    Messages:
    27
    Likes Received:
    1
    +1 here. Tunnel monitoring for all SBC's would make sense.
    And communication! At least an e-mail to all partners as soon as the first few complaints came in.
    I knew all about the live webinar in the middle of the night when the new version came out, so it's not that you guys don't know how to reach us!
     
    Nathan Boyd likes this.
  14. uptime1

    uptime1 New Member

    Joined:
    Jan 13, 2012
    Messages:
    112
    Likes Received:
    28
  15. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    Another SBC failure today and again I had no idea until I noticed the unregistered phone. This is with the new SBC 15.5.2607.1 that fixes the certificate expiration issue. Rebooted the server and when the SBC came back up, it reported
    Code:
    ERR | 20170629-085735.791 | 3CXTunnel | RESIP:TRANSPORT | 1348 | InternalTransport.cxx:148 | Could not bind to [ V4 192.168.1.100:5060 UDP flowKey=448 ]
    ERR | 20170629-085735.791 | 3CXTunnel | TUNL | 1348 | Bridge.cpp:165 |
    ** Resip exception caught while starting bridge '3CXSBC15.5.2607.1' (123456)
     ! Exception: Could not use port
    There's no reason that 5060 should be in use. Restarted the SBC and it worked fine. But for this thread, the point is, the SBC can fail for many reasons; we need notification so we can fix it.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. OCWI

    OCWI New Member

    Joined:
    Jan 17, 2017
    Messages:
    159
    Likes Received:
    46
    Mark,

    Im curious, was this on a windows SBC or a Raspberry Pi?
     
  17. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    171
    Likes Received:
    14
    Windows Server 2012 R2.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...