Issues with SBC - "Too long inactivity" on TCP connection every few minutes

[codyj]

Hi All -

Running into an issue with a new deployment of 3CX SBC on Debian 9 that we are testing in a remote office. The tunnel seems to establish itself ok, and even logs that it is sending and receiving keep-alives, but every few minutes the tunnel disconnects. The log below is from my 3cxsbc.log in debug mode. You can see the keep alives being sent and received, but the statistics after the tunnel disconnects show that the last keep alive was received 20271 ms ago even though the log shows a keep alive was received just before the disconnect. Has anyone seen anything like this before, or know where my issue may be? Where you see the xxx.xxx.xxx.xxx in the log, our public IP address is there, I've just removed it before posting the log here.

Any help or ideas would be very much appreciated!

debug:3CXTunnel[734]::2018-05-04T11:24:42.356609-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:523 | UDP channel is acknowledged

info:3CXTunnel[734]::2018-05-04T11:24:42.627315-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/security.cpp:1342 | Keep-alive received

debug:3CXTunnel[734]::2018-05-04T11:24:43.358011-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 238, KA recv: 54

debug:3CXTunnel[734]::2018-05-04T11:24:44.359303-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 239, KA recv: 54

debug:3CXTunnel[734]::2018-05-04T11:24:45.360588-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 240, KA recv: 54

debug:3CXTunnel[734]::2018-05-04T11:24:46.361902-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 241, KA recv: 54

debug:3CXTunnel[734]::2018-05-04T11:24:47.363187-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 242, KA recv: 54

info:3CXTunnel[734]::2018-05-04T11:24:27.481536-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/security.cpp:1342 | Keep-alive received

debug:3CXTunnel[734]::2018-05-04T11:24:27.483298-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:765 | Sending keep-alive. Stats: KA sent: 243, KA recv: 55

debug:3CXTunnel[734]::2018-05-04T11:24:27.484663-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:506 | Sent UDP channel re-check request sbc.c35e8dd8$8844 to [ V4 xxx.xxx.xxx.xxx:5090 TCP ]

debug:3CXTunnel[734]::2018-05-04T11:24:47.752117-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:523 | UDP channel is acknowledged

err:3CXTunnel[734]::2018-05-04T11:24:47.753515-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:268 | Bridge [3CX Session Border Controller] failure 'Too long inactivity' on TCP connection: while processing tunnel connection

debug:3CXTunnel[734]::2018-05-04T11:24:47.754718-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/TunnelTcp.cpp:581 | === Statistics:#012  Keep-alives: recv 55, sent 243#012  Transfers:   recv last 20271 ms ago,#011total 13565 bytes#012               sent last 1 ms ago,#011total 18826 bytes#012===

warning:3CXTunnel[734]::2018-05-04T11:24:47.756162-05:00: TUNL | 139853381245120 | /home/repomaster/workspace/SBC/Sources/Projects/3CXSBC/Bridge.cpp:261 | Tunnel disconnected. Terminating all active calls.

 

[eddv123]

Hi Codyj,

What phone devices are you using across the tunnel ? are they fully supported by 3CX ?
https://www.3cx.com/sip-phones/

Do they appear in your phones list with :SBC on them ?

If these disconnections are random you might want to check your firewall checker posts green for SIP ALG also. You should have port 5090 enabled as the tunnel would not connect at all.

Is your broadband fairly good ?

 

[Vali_3CX]

@codyj
"the statistics after the tunnel disconnects show that the last keep alive was received 20271 ms ago even though the log shows a keep alive was received just before the disconnect."

Not exactly, look again:
- last received keep-alive (#55) was at 11:24:27.483298;
- statistic is shown at 11:24:47.754718, aprox 20 seconds later (i.e correctly 20271 ms )
- statistic correctly points there were 55 keep alives received
20 seconds is not exactly "before the disconnect"

 

[codyj]

Hi Codyj,

What phone devices are you using across the tunnel ? are they fully supported by 3CX ?
https://www.3cx.com/sip-phones/

Do they appear in your phones list with :SBC on them ?

If these disconnections are random you might want to check your firewall checker posts green for SIP ALG also. You should have port 5090 enabled as the tunnel would not connect at all.

Is your broadband fairly good ?

When testing earlier, there was a Yealink T27G and an Aastra 6735i that were both using the SBC. I've removed the Aastra phone from the equation, and now only have the Yealink T27G using the SBC and am watching the logs again.

I did not have the T27G provisioned thru 3CX before, I had just programmed the settings manually so it showed in 3CX as "UNPROVISIONED". I've added the T27G to the extension now and it is showing IP as "via SBC" now.

I know 5090 is allowed, but I'll check on the SIP ALG. Thank you.

Broadband should be sufficient. 50/20 on the SBC end, and 100/100 at the 3CX end. We currently run our devices remotely without the SBC with no issues. We just want to add-in the SBC to manage those connections more efficiently.

@codyj
"the statistics after the tunnel disconnects show that the last keep alive was received 20271 ms ago even though the log shows a keep alive was received just before the disconnect."

Not exactly, look again:
- last received keep-alive (#55) was at 11:24:27.483298;
- statistic is shown at 11:24:47.754718, aprox 20 seconds later (i.e correctly 20271 ms )
- statistic correctly points there were 55 keep alives received
20 seconds is not exactly "before the disconnect"

You may have discovered something here. You're correct, I missed that the last keep alive received was at 11:24:27, however if you look above that there was a keep alive received at 11:24:42. It's almost as if there's a timing issue here somewhere, and when the SBC received a keep alive from 20 seconds earlier it made it think that the tunnel disconnected. Any thoughts?

 

[codyj]

It dropped again after 845 keep alives sent and 202 received. Once again, the last keep alive received had a time stamp from the past, so this definitely could be the issue. Unfortunately I don't know what would be causing it though. Screenshot attached.

 

[Vali_3CX]

SBC Tunnel has its internal client connection timeout (20 secs). If it don't receive a keep-alive packet from the client during this interval, it assumes connection lost and disconnects.
If you look in the attached screenshot, again:
- last keep alive #202 has been received at 14:10:22.323784
- stats+disconnection at 14:10:42.560320 (20 secs later, 20236 ms)

 

[ivank]

Hi codyj, I have a question.
Have you enabled encryption during SBC installation?
If not, try setting "SecurityMode = 1" in /etc/3CXSBC/3cxsbc.conf and restart the SBC.
If it starts to work without problems, please tell me.

 

[codyj]

Hi codyj, I have a question.
Have you enabled encryption during SBC installation?
If not, try setting "SecurityMode = 1" in /etc/3CXSBC/3cxsbc.conf and restart the SBC.
If it starts to work without problems, please tell me.

I did have it enabled originally during installation but had changed it to 0 as I read about that possibly being an issue. Unfortunately it didn't fix the issue.

 

[ivank]

Have you tested the connection quality using 3rd party tools (from both sides, SBC and PBX)?
Is SBC's host used for 3CXSBC exclusively?

 

[loyer]

I was having issues with the SBC a few months ago with a client and it ended up being too much packet loss at their site. Almost everything looked normal and great but continued to get errors and dropped calls. I used this VoIP test site to discover the packet loss: https://www.voipreview.org/speedtest I ran it from the server that SBC was installed on. The packet loss ended up being from a firewall/router that was old and was under performing and dropping packets. The client bought a new firewall and it was so much better.