Tunnel instability between 3CX server and 3CX SBC

[lanb]

Hi,
I have some reconnections between the 3cx server and the 3cx sbc, about 1 to 4 times per day.
Each time, the server, or the SBC, complains about a keep-alive timeout, and stop or restart the tunnel.
We could think that it could be a network lag, but I managed to find that during this time, datas are streamed between SBC and server.

I ran some tests to check what is faulty, and it looks like the 3CXTunnel process may lag.
If I log "netstat" output every second, I can see that before the timeout, the recv-q of the 3CXTunnel increase, and stay not empty for some seconds, and finally hit the timeout.
I logged also "top" output to see if a process takes all the 2 cpu time, but no. And meanwhile, all other process are running fine, even my log process.

I have lots of free memory, look at the current top header :

top - 17:20:29 up 174 days, 23:38,  1 user,  load average: 0.52, 0.39, 0.31
Tasks: 111 total,   1 running, 110 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.9 us,  5.6 sy,  0.0 ni, 88.2 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem : 12017472 total,   559260 free,  1423456 used, 10034756 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used. 10015900 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                 
 6329 phonesy+  20   0 3685644 357432  80376 S   5.6  3.0 883:28.57 3CXManagementCo                                                                                                                                                                                                                                         
28043 phonesy+  20   0 1205220  27568  10016 S   5.6  0.2   2747:42 3CXMediaServer                                                                                                                                                                                                                                           
21323 phonesy+  20   0 1247112  51136  14204 S   3.0  0.4 258:54.52 3CXIVR                                                                                                                                                                                                                                                   
21102 phonesy+  20   0 1263368  73404  14280 S   1.7  0.6 140:52.88 3CXPhoneSystem                                                                                                                                                                                                                                           
21429 phonesy+  20   0 1023892  16340   9484 S   1.3  0.1 274:33.62 3CXTunnel                                                                                                                                                                                                                                               
28091 phonesy+  20   0 3414444 149388  53024 S   1.0  1.2   1226:48 CloudServicesWa                                                                                                                                                                                                                                         
 6317 phonesy+  20   0 1039056  68424   9300 S   0.7  0.6  96:27.58 3CXSLDBServ                                                                                                                                                                                                                                             
 6318 postgres  20   0  289724  22948  19144 S   0.7  0.2  20:23.72 postgres

I have collected SBC and server logs if it can help.

3CX Version : Enterprise 16.0.4.493
Server OS : Debian 9
3CX Server is in a VM
SBC version : 16.0.390
SBC OS : Debian 9
SBC is in a VM

Is it a problem of the 3CXTunnel process ?
If not, how can I troubleshoot this deeper ?

Thanks for your help

 

[AWS2P]

Hi, don't know if it can be same problem as i encountered with bridges, with no explainations for now.
https://www.3cx.com/community/threads/bridge-behaviour-pbx-v16-sp4.70878/
https://www.3cx.com/community/threa...o-master-when-using-tunnel.70524/#post-312128

 

[leejor]

I don't suppose that you have a second SBC, at another location, to use as a comparison. It it had errors, then I'd suspect an issue at the server end, if it ran with no issues, then perhaps it is the network, or other issue, at the first SBC. Do you use any firewalls? Did this issue just start, or has it always been there?

 

[lanb]

@aws2p : interesting because your problem raises at the same frequency that mine. I think that I saw same logs with the standard text "DNS error resolving or service unavailable" when the connection is shut down by the server. In the verbose log files, it appeared to be a keepalive timeout.

@leejor : Yes I have only one SBC. But your idea is good to build another one, from another network to check if the second tunnel goes down at the same time.

 

[AWS2P]

in my case, in the time brigde was down with tunnel 5090 port , nothing was wrong with trunks or with SBC using also tunnel 5090 port but connected to OVH cloud pbx

 

[cobaltit]

@lanb

It would help to clarify what VM platform is on and where as well as what VM platform your SBC is on and where. For example we run our 3CX instance in Azure with SBCs on a Raspberry Pi 2 and HyperV VM with none of the issues you describe.

 

[lanb]

The server is on a VPS cloud ram 2 at OVH (2 cpu, 12GB ram, 50 GB SSD) Gravelines
SBC is under proxmox. I manage this physical server that is in our building. The VM has 2 cpu, 1GB ram, 20 GB SSD. The building is connected by a dedicated optical fiber, 100Mb symmetric, and some spare connections. When the problem occurs, the traffic is very low, every time under 5 Mb.

Does the uptime of the server (174 days) can be a problem ?
I can try to restart it tomorrow to check this.
I will add another SBC also, to check if the two connections go down at the same time.

 

[JohnS_3CX]

Is your SBC in a KVM or LXC under Proxmox?
Officially only KVM is supported (along with some others)
https://www.3cx.com/docs/manual/phone-system-installation-windows/#h.5hoy5wwhhk5c

Also you need to check the SBC requirements to make sure the SBC is not overloaded
https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/#h.za60f8obhoix

There should be nothing else installed or running on that machine. Use our Debian ISO if you are not sure about the integrity/compatibility of the OS https://www.3cx.com/phone-system/download-links/

If these basic requirements are not met, don't spend any time in troubleshooting further. Start from a supported scenario and build your way up accordingly

 

[lanb]

Thank you for your help @JohnS_3CX

SBC is in a KVM
this VM is alone running in proxmox, the other VM are down.
SBC is installed as explained in this page, in debian chapter : https://www.3cx.com/docs/3cx-tunnel-session-border-controller/
There is no foreign process running on SBC or 3cx server. I just added a crontab to upload the backup made during the night.

The requirements are not completely met for SBC, as the VM has only 1GB ram and only 20GB disk, but the SBC never hit the limit of memory and never swap (and the disk is far from full). Behind the SBC, there is only 5 devices. Some of them have extensions, so if I sum all the BLF of all the devices, I can count a maximum of about 150 BLF behind this SBC.

The question will be cleared with the new SBC that I will install tomorrow, with all the memory and necessary CPU (and no device behind it).

I come back in 3 days to tell what happened after reboot, and add of the isolated SBC.

 

[JohnS_3CX]

Ok keep us posted of your progress. Please note that if the machine is not overloaded, and the tunnel still drops, you will have to start looking into networking issues.

The tunnel will disconnect usually when the network conditions are unreliable. Unreliable can mean connectivity issues, but it could also mean that the firewall may be misbehaving or being too aggressive in the way it manages connections. The suggestion @leejor made is a good way to see if the issue is on the server side or the SBC side. Setup and SBC elsewhere (i.e at home or somewhere with a less strict network) and see if it also drops.

 

[lanb]

It seems to be a problem of our cloud operator with overloaded host.
Thanks

 

[JohnS_3CX]

You're welcome, hope they help resolve it.