New full company rollout - persistent audio problems

[CKFIT]

Hi, everyone! This is my first post here, so I'm sorry if I inadvertently leave any important info out of this initial post.

We just rolled out a 3cx system to our company that has offices in 4 different cities. We have had numerous users state that after using their phones successfully for days, all of a sudden they can not hear incoming audio, but the other person can hear them. This seems to coincide after local internet outages (or potentially power-related network power cycles) where the location fails over to a backup internet connection. After a RIDICULOUS amount of troubleshooting (numerous phone reboots, factory resets, firmware re-flashes, extension re-creation ect), it seems like the only thing that fixes the phone is simply leaving it unplugged for roughly 10 minutes and powering it back up.

After literally hours of troubleshooting with numerous users, it's kind of nice to know that the easy fix is to simply have them unplug the phone for 10 minutes and plug it back in, but it would be nice to know exactly what is causing the issue so we can fix it or come up with an immediate fix when a user lets us know it's happening to them.

Any ideas? Thanks in advance!

Server: Windows Server 2016 in Azure
Phones: Fanvil x3, x4 (x6, x7, x210 have not been affected, but far fewer of these are implemented company-wide)
Provisioning: STUN

 

[FS_Mathieu]

Hello @CKFIT !

Do you know how many phones provisionned by stun there is behind each firewall ? It looks like a firewall issue more than phone issue since phones models use different sip and rtp ports !

 

[Evolute IT]

Hello @CKFIT !

Do you know how many phones provisionned by stun there is behind each firewall ? It looks like a firewall issue more than phone issue since phones models use different sip and rtp ports !

If more than 3, use the SBC. Fixes most issues.

 

[JohnS_3CX]

Hi,

it looks like the ports are closed by the firewall after a while. A reboot of the phone probably reopens them and restores the audio.

The SBC would indeed help in this case, otherwise if youonly use a couple of phones per sitw you can make some static port forwards for those devices on the firewall for both the SIP port (which must be unique per device) and the audio ports (which must be non overlapping across devices per each site).

The SBC saves you the trouble of maintaining ports and helps you overcome NAT problems easily compared to doing it manually.

 

[Saqqara]

As above, if you have more than 3 stun configured phones per site, go down the SBC route.

If you want to stick with the stun route, you need to do the following

1) In the Management Console each Remote STUN Extensions must have a unique Local SIP Port and Unique RTP Port range and the RTP Port range must be 10 ports (for Yealink phones 12 ports) per remote site. This means that if we only had Yealink phones that would be e.g.:

• Ext A should have Local SIP Port: 5065 and RTP Port Range: 14000-14011 (12 ports)
• Ext B should have Local SIP Port: 5066 and RTP Port Range: 14012-14023 (12 ports)
• Ext C should have Local SIP Port: 5067 and RTP Port Range: 14024-14035 (12 ports)
• Ext D should have Local SIP Port: 5068 and RTP Port Range: 14036-14047 (12 ports)
• etc...

2) Once you have done this, then on the Remote Location Router/Firewall behind which the phones are residing, you must:

• Disable SIP ALG or any similar port remapping feature.
• Create NAT/Port Forwarding Rules so that, as per the above example:

--- Ports 5065 TCP/UDP and 14000-14011 UDP are forwarded to the IP of Ext A
--- Ports 5066 TCP/UDP and 14012-14023 UDP are forwarded to the IP of Ext B
--- Ports 5067 TCP/UDP and 14024-14035 UDP are forwarded to the IP of Ext C
--- Ports 5068 TCP/UDP and 14036-14047 UDP are forwarded to the IP of Ext D
--- etc...
- This also implies that the the phones/endpoints should have Static LAN IPs assigned to them.

1. Reprovision all Phones.
2. For all Remote STUN Extensions, go into the Management Console --> Extensions, edit the Extension, and in the "Options" tab, check the "PBX Delivers Audio" option, then press OK.

 

[CKFIT]

Thanks everyone for the responses.
We have all of our phones (196) as STUN because the PBX is on the Azure cloud.
We'll go ahead and configure a SBC on each site. I'm assuming we have to change all extensions to remote SBC instead of STUN. Will this be seamless to the user meaning we don't have to remove and re-add all of the fanvil's to the PBX, correct (done afterhours, of course)?

 

[FS_Mathieu]

We have all of our phones (196) as STUN

So you have something like 50 phones behind each firewall ?

W
T
F

So you have 50 SIP Rules and 50 RTP rules on each firewall ?

Will this be seamless to the user meaning we don't have to remove and re-add all of the fanvil's to the PBX, correct (done afterhours, of course)?

I think you'll only need to switch to SBC and put the IP of the SBC in each ext

We'll go ahead and configure a SBC on each site

You'll need SBC Performances

 

[JohnS_3CX]

Also consider that in case one STUN phone gets blacklisted, any other phones under that office IP might also be affected.

Yes they will have to be re-added to ensure they load the correct configuration (meaning you will switch the provisioning to SBC under each extension's devices and put in the SBC IP).



After adding your SBCs you will have to reprovision all the phones so there will be a bit of downtime. The SBC can coexist on the network with the existing STUN-configured phones so those phones will continue to work via STUN until it's their turn to be rebooted and reprovisioned.

Edited: for more clear procedure

 

[JohnS_3CX]

See edited message above for more clarification

 

[CKFIT]

Lol. No, the max is 30 right now behind a firewall but didn't have to configure rules for each phone as outbound rules allow those ports for all the subnet. We have 1Gb fiber at all locations so we haven't had performance issues on the live locations.
We whitelisted all of our locations as we did get blacklisted on the beginning. It's been a learning process coming off an old Toshiba system.
The other site with about 120 will go live tomorrow so we have time to get them reconfigured. I already installed the SBC on this large site and moved a couple from STUN to SBC and just reprovisioned them and worked fine, no problems.
Thanks for all of the info!

 

[Saqqara]

Lol. No, the max is 30 right now behind a firewall but didn't have to configure rules for each phone as outbound rules allow those ports for all the subnet.

This is why you are having issues, you have not configured the phones correctly.

 

[Saqqara]

As you have 1GB connection at each location, you could setup site to site vpn between the office and Azure.

This way you configure the phones as local, you would have to setup DHCP option 66 - https://www.3cx.com/sip-phones/dhcp-option-66/

 

[Mark Phillips]

As you have 1GB connection at each location, you could setup site to site vpn between the office and Azure.

This way you configure the phones as local, you would have to setup DHCP option 66 - https://www.3cx.com/sip-phones/dhcp-option-66/

I would thoroughly recommend Azure site to site VPNs if I had to worry about 4 sites and c.200 extensions.