Issue with SBC and all phones

[robausirl]

Hi All,

Today I had a weird issue. I have a 3CX server running latest stable release in the cloud and an on-premise SBC. Only using Yealink phones and out of no where all phones popped up with no service, at almost the same time I received an alert from 3CX stating that the SBC was up... Normally I see an alert stating it is down and then up but not this time.

I could see all phones in the web portal but they were showing as new and ready to be assigned. I checked each user and confirmed they were already assigned... Weird.

- Restarted all 3CX services - No fix
- Rebooted SBC - No fix
- Rebooted all phones - No fix

Finally I deleted all phones from 3CX and re-added them to each user and boom they started working again.... Very strange.

From looking over the logs around the time I got the alert i can see the below. Can anyone help me understand what went wrong and how to prevent it from happening again?

10:43:47.797|7f5846919700|Error|SipCore.cpp(708): OK w/out SDP: SipResp: 200 tid=6a23b671b2f36a4d cseq=10214 INVITE [email protected]:5060 / 10214 from(tu)
10:44:25.476|7f5846919700| Info|TCPSide.cpp(300): 37<-::ffffCLIENT WAN IP REMOVED FOR PRIVACY):37021:22: Destroyed; num.conn=1
10:44:25.476|7f5846919700| Info|TLSTransp.cpp(436): TLS broke
10:44:25.476|7f5846919700| Info|Tunnel.cpp(212): Destroying Tunnel 123456
10:44:25.476|7f5846919700| Info|Tunnel.cpp(58): Reporting disconnect of sbc.a630532c.1YRyhgBsWChv from ::ffffCLIENT WAN IP REMOVED FOR PRIVACY)
10:44:25.476|7f5846919700| Info|Tunnel.cpp(331): Forcing termination of (SBC AUTH KEY ID REMOVED FOR PRIVACY)
10:44:40.034|7f58426ff700| Info|Stack.cpp(390): SIP Stats:

 

[ChrisC_3CX]

just so that we first get a better understanding of your current setup, please provide the relevant information mentioned here:
https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

Also, please provide information regarding the 3CX SBC:

• Which exact version of the 3CX SBC are you running(check SIP Trunks >> Edit SBC >> Statistics >> Build version)?
• What machine is the 3CX SBC hosted on (OS and Hardware specs)?
• Have you got more than this one SBC connected to the 3CX Instance? Was that affected as well?
• Is there any chance the local IP of the 3CX SBC changed when the issue occurred?

 

[robausirl]

• 3CX Version = Pro Annual 16.0.9
• Server OS = Debian 9
• Is the 3CX Server Hosted and where? = AWS
• IP Phone Make/Model/Firmware = Yealink T53
• Provisioning Method = SBC
• Trunk Provider or Gateway Make/Model = Vocus 2talk
• Has the Firewall Checker passed = Yes
• Are custom Phone Templates being used = NO as it's an SBC setup
• SBC is on a Raspberry Pi 4 4GB with only 5 phones behind it and running Raspian. Version says 16.2.24
• Only the one SBC connected
• No chance on IP change as it's static and IP is the same now as the day I put it in

 

[ChrisC_3CX]

Hey @robausirl

Thanks for the info. All does look well regarding the configuration so this will definitely require a more in-depth investigation preferably with the help of our Support Team.

However, if this is not a re-occurring issue or generally not a pressing matter you might first want to investigate a bit further yourself. Based on the provided logs it seems the tunnel connection with the 3CX SBC was terminated. This could occur for a number of reasons one of which is no activity on that connection for a certain period of time. That said, I'd recommend also looking at the Pi's system logs in order to determine if anything worth noting happened during that time that may have caused the issue (Pi reboot, services dropping, high resource usage, network outage, IP conflict etc).

Do note that, whatever happened probably lasted less than 5 minutes as that would explain why you never got a notifications saying that the SBC was down and instead got one that just said it was up. For the "SBC down" notification to be sent it needs to be in a disconnected state for at least 5 minutes.

 

[robausirl]

Thanks for the replies. I did not know the 5 minute window so that is helpful.

I did do some further digging in the logs and can see that around the time I attempted a services restart on the 3CX server that the below was logged. Why would the ID not match up and how can i explore this further?

12:39:21.258|7f584ab2c8c0| Warn|Bridge.cpp(466): Currently mapped connection with id=sbc.a630532c doesn't match to one being removed, ignore removal

Also where can i find the SBC logs on the PI, as in location and i'll dig a little deeper there as well?

 

[ChrisC_3CX]

If the message you mention was seen during the time you restarted the 3CX Services, it most likely has nothing to do with the issue you experienced.

The 3CX SBC logs(3cxsbc.log) can be found in /var/log/. Do note however that when I mentioned the logs on the 3CX SBC I actually meant the system logs of the OS hosting the 3CX SBC that also can be found in /var/log. If you can't find any indication of what might've happened by investigating the OS and network level, you can of course also review the 3CX related logs if you want, however, if it does come to that, I'd recommend taking this up with our Support Team for further investigation

 

[robausirl]

Just checked those 3CXSBC logs and found the below error. Although I don't see why my connection would be refused. What does this error mean exactly? SBC registration was refused by the 3CX server.... Seems strange as the server shows nothing in it's logs about it... The system logs don't show anything at all around this time.... This is super weird...

Bridge failure 'Connection refused' on TCP connection

If the issue does re-occur I will log a support ticket but thought i'd ask on here to deepen my knowledge in case anyone else has come across any of this in the past as my setup is fairly generic and i've got other clients running the same setup without issue.

 

[ChrisC_3CX]

I'm afraid the error by itself will not be enough to identify the exact cause of the issue as there are many factors that come into play. That is why I recommended contacting 3CX Support as a thorough investigation is in order if we are to pinpoint the issue's origin or at least narrow it down.