Calls Randomly Dropping and Phones Un-Register / Re-Register Often

[dmaynard71]

I'm having issues with calls dropping, seemingly at random, after just a couple minutes of call time. Call quality is good, the call just disconnects. I've been unable so far to isolate if any particular extensions are doing this.

In looking for possible reasons, I noticed that many of the extensions are un-registering / re-registering pretty frequently. When they unregister, they will reregister anywhere from 1 second, up to 2 minutes later. Most of the time they re-register a second or two later. Some will un-register again within a minute or two. The worst example I've found is a an extension that cycled un-reg/reg 11 times in a 20 minute period!

Can a phone un-register during a call? Would that cause the dropped calls? I have no idea if these two behaviors are linked or not, but I need to fix both,

Setup Basics:
3CX running on Google Cloud
3CX SBC running on a dedicated Win 10 workstation on LAN
16 Fanvil X4G phones on LAN, never more than 3-4 in use at once.

 

[Evolute IT]

Please provide the following informations:

• 3CX Version, e.g. Standard Annual 16.0.2.910
• Server OS, e.g. Debian 9 / Windows Server 2012 R2 / Raspberry Pi
• Is the 3CX Server Hosted and where?
• IP Phone Make/Model/Firmware
• Provisioning Method: Local / VPN / STUN / SBC
• Trunk Provider or Gateway Make/Model
• Has the Firewall Checker passed: YES / NO
• Are custom Phone Templates being used: YES / NO

 

[eddv123]

Can a phone un-register during a call?

This is purely circumstantial. It can de-register if there is an environmental change that cuts connection from endpoint to registrar, such as loss of internet or firewall closing the connection (like when SIP ALG is enabled). https://www.3cx.com/community/threads/calls-end-at-32-seconds.64231/

The Yealink devices which we use (and I am sure it is the same for any other) is aware that there is a call in progress and for example like this test I did will pull up an error if any changes are made (within the boundaries of its own settings) and pull up the attached error.

 

[JohnS_3CX]

Phones will usually refresh their registration at regular intervals at 120sec

So every 20 times seeing 10 registrations should not really be a problem.

Please time the calls and see when they drop to see if they coincide with SIP timers

 

[dmaynard71]

Please provide the following informations:

• 3CX Version, e.g. Standard Annual 16.0.2.910
• Server OS, e.g. Debian 9 / Windows Server 2012 R2 / Raspberry Pi
• Is the 3CX Server Hosted and where?
• IP Phone Make/Model/Firmware
• Provisioning Method: Local / VPN / STUN / SBC
• Trunk Provider or Gateway Make/Model
• Has the Firewall Checker passed: YES / NO
• Are custom Phone Templates being used: YES / NO

• 3CX Version, Standard Annual 16.0.493
• Server OS, debian-9-stretch-v20191115
• Is the 3CX Server Hosted and where? Google Cloud/Compute Engine
• IP Phone Make/Model/Firmware Fanvil X4G - FW. 2.10.1.6836
• Provisioning Method: SBC Build 16.0.390
• Trunk Provider Vitelity
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

 

[dmaynard71]

Phones will usually refresh their registration at regular intervals at 120sec

So every 20 times seeing 10 registrations should not really be a problem.

Please time the calls and see when they drop to see if they coincide with SIP timers

I am not able to visit the site, but it is being reported that calls will drop after anywhere from a few seconds to multiple minutes. The system is considered completely unreliable at the moment.

 

[dmaynard71]

Ok, I have a clue for you folks that seems possibly relevant.

I was watching the statistics tab for the SBC on the 3CX Management Console and the SBC uptime changed to 1 minute. Almost immediately after, someone told me their call just dropped.

I know very little about expected SBC behaviour, but I would think uptime should have to be very stable?

Currently shows: Register sent (up arrow) 14 min, Register OK 1-17-2020 11:58:48, Last Failed Register 1-17-2020 10:58:54

EDIT: Just checked again: Register sent (up arrow) 3 min, Register OK 1-17-2020 12:15:46, Last Failed Register 1-17-2020 10:58:54

EDIT 2: Another check: Register sent (up arrow) 0 min, Register OK 1-17-2020 12:32:22, Last Failed Register 1-17-2020 10:58:54

What would some possible causes of the SBC not staying registered be? Or am I even on the right track?

 

[dmaynard71]

Followup on post #7 from yesterday.

No one is in the office today, so I placed several calls and just let the line sit open while watching the registration for the SBC. The moment the SBC re-registers and uptime goes to zero, the calls drop.

I guess I've found the problem, but what I don't have a clue about is why the SBC wants to re-register so often. It does it randomly, anywhere from a minute or two, up to about 32 minutes is the longest I've seen. 3-8 minutes seems most common.

 

[eddv123]

Try running the verbose level logging from the server and push it down to the SBC device from the PBX >> Replicate the fault >> Check the logging.

The closest 2 issues I have had like this were due to the following (however the logs should say quite clearly anyway):

* Local DNS server issues - if the local site does not have reliable DNS resolution this can cause a lot of issues with the tunnel the SBC uses - a drop in DNS = a drop of the tunnel.

* Multiple firewalls used: I had a site where the SBC was located behind multiple layer 3 devices (put in after install of the system and SBC without notice to us. One of these devices was closing the TCP connection from the SBC (it could be seen in the logs) as for some reason it did not think the connection was up any longer.

Another thing you could check on the SBC config is if you are using TCP or TLS for your transport type (under settings >> options >> security), alternating to the opposite may help too.

 

[dmaynard71]

eddv123 said:

Try running the verbose level logging from the server and push it down to the SBC device from the PBX >> Replicate the fault >> Check the logging.

Oof, I hardly know what I am looking at! I am finding entries that say "Terminating all active calls" and "Tunnel disconnected" that correspond to the disconnect times. So, I'm close but am unable to sort out the WHY part.

The closest 2 issues I have had like this were due to the following (however the logs should say quite clearly anyway):

* Local DNS server issues - if the local site does not have reliable DNS resolution this can cause a lot of issues with the tunnel the SBC uses - a drop in DNS = a drop of the tunnel.

In logs: I'm seeing a lot of this, so I guess DNS is resolving ok? Possibly that doesn't mean it's reliable. Not sure on this at all.
INFO | 20200118-110744.389 | 3CX | SBC | 13684 | dns.cpp:89 | [DNS] Resolving IP address(es) for MY_FQDN
DEBUG | 20200118-110744.389 | 3CX | SBC | 13684 | dns.cpp:109 | [DNS] Found in cache
INFO | 20200118-110744.405 | 3CX | SBC | 13684 | dns.cpp:123 | [DNS] Resolved host name MY_FQDN to [MY_3CX_IP]

I had a thought to change the 3cxsbc.conf file so it uses the IP address for the PBX instead of FQDN, but I don't know if that would solve anything, plus I haven't sorted out how to do that yet.

* Multiple firewalls used: I had a site where the SBC was located behind multiple layer 3 devices (put in after install of the system and SBC without notice to us. One of these devices was closing the TCP connection from the SBC (it could be seen in the logs) as for some reason it did not think the connection was up any longer.

I believe the SBC is only behind an edge router/firewall of some kind and one switch. I don't have access to the physical equipment, or login credentials for those at this time. What can I look for in the logs?

Another thing you could check on the SBC config is if you are using TCP or TLS for your transport type (under settings >> options >> security), alternating to the opposite may help too.

Switched from TLS to TCP. Not a solution that I can tell. Uptime is longer between disconnects (10-44 minutes so far), but that may be a coincidence.

 

[eddv123]

If its a DNS issue you get an error to say so - want to PM me the full logging ?

 

[dmaynard71]

If its a DNS issue you get an error to say so - want to PM me the full logging ?

Yes, please. I'll PM you shortly.

 

[eddv123]

Ive had a brief look over your logging and responding via the string as opposed to direct so other users may benefit from this information.

I think it is a transport error since first your switch to TLS seems to be successful, and in the logs I am seeing:

WARNING | 20200118-114850.385 | 3CX | SBC | 13684 | bridge.cpp:143 | Closing tunnel connection
NOTICE | 20200118-114850.385 | 3CX | SBC | 13684 | tunneltcp.cpp:311 | Closing TCP socket
NOTICE | 20200118-114850.385 | 3CX | SBC | 13684 | security.cpp:1162 | TCP Accepted -> Invalid

This is what I saw when that firewall I had was closing the TCP connection of my SBC, I think you have the same here.

 

[dmaynard71]

Switched from TLS to TCP. Not a solution that I can tell. Uptime is longer between disconnects (10-44 minutes so far), but that may be a coincidence.

I realized after this statement that I had not actually switched from TLS to TCP at that point (the change wasn't applied properly). I have now changed correctly and verified that the SBC is using TCP.

I am currently at just over 4 hours uptime since exactly when the change was applied, and I held a couple calls open for over an hour that didn't drop. I feel reasonably satisfied that this is the issue, but I will monitor the uptime over the weekend and Monday. I'll try to remember to post the result and if I'm satisfied this resolved the issue for me.

Thanks everyone, especially edvv123, for your help.

 

[dmaynard71]

SBC has been stable and up for 25+ hours now. WHooo Hoooo!

Just a final note regarding the thread title, the un-reg/dereg thing wasn't really the issue here, it was the dropped calls. The calls were dropping due to unstable SBC connection. The fix was to change security type for the SBC on the PBX side and then restart services of the local SBC.

I am assuming case closed at this point.

 

[JohnS_3CX]

Hello,

Please continue to monitor this and see if it comes back.

The fact that the SBC disconnects while in TLS mode but doesn't while in non-TLS mode is a sign that something is affecting TLS transport. You may have too look into the networking side of things a bit further, perhaps one of the edge devices is getting in the way?

 

[dmaynard71]

Hello,

Please continue to monitor this and see if it comes back.

The fact that the SBC disconnects while in TLS mode but doesn't while in non-TLS mode is a sign that something is affecting TLS transport. You may have too look into the networking side of things a bit further, perhaps one of the edge devices is getting in the way?

Good catch, thank you for considering that. I hope to be able to do that soon. As for now, all is working and everyone is happy.