Provisioning and updates - issues with new router

[InSite TJW]

We have a 3CX PBX in 'the cloud', a session border controller (SBC) running on a Linux Virtual Machine on-site, a large number of Fanvil X3G IP phones and four Yealink CP920 conference phones. We recently moved to new premises - before the move everything seemed to be working. Now we seem to be unable to provision the CP920s so have been forced to use the standard X3G phones in the conference rooms as a temporary measure. We have recently taken delivery of a new batch of X3G phones with the factory firmware version 2.4.0.5487 and I am unable to provision these too. If I manually upgrade the firmware on the new X3Gs using the phone's web interface and new firmware downloaded from 3CX I can provision them, but in the old building I was able to provision a batch of factory-firmware phones then update the firmware in a batch using the 3CX management console. That batch upgrade saved a fair bit of time last time a firmware update was released, but I missed three phones which are still on (3CX-branded) 2.10.0.6586; if I select these and request the firmware update from the Management Console they reboot but don't install the update - in the old building that would have worked.

The only big change in our environment is the router, in the old place internet access was provided as part of the office lease. We now have a router ordered through the ISP. I therefore suspect there is a router configuration issue but I have no idea what or how to go about identifying what needs to be changed. The only other change we have made is the network address used by the phone system (it was 10.80.22.0 255.255.255.0, the subnet mask is unchanged but the addresses start with 172).

Please help us work out why we are having these issues.

 

[JohnS_3CX]

Hi @InSite TJW

Any phones that do not have supported firmware must be manually upgraded as per our procedures (can be found here https://www.3cx.com/sip-phones/ ). The future firmware upgrades should work fine after the initial one is brought up to date.

Phones that have newer firmware than what we support might not be able to provision at all as you have noted, until you manually downgrade the firmware to the correct version.

Phones on a very old firmware that won't upgrade automatically (while other phones on the same network do so just fine) you might be in a situation where you need to step-upgrade those phones manually. Meaning you must visit the manufacturer's site, and download all the firmware versions that came after the one you are running now, all the way up to just before the latest 3CX version from our own site.

All the above aside now, if you generally speaking lost the ability to upgrade firmware via 3CX then you will need to check your PBX-defined HTTP and HTTPS ports are reachable by the phones (thats where they will actually download the ROM file from). A quick way to see of the firmware is downloadable:

1. Open your provisioning folder to see the name of the rom file here: C:\ProgramData\3CX\Instance1\Data\Http\Interface\provisioning\uniqueID\firmware\yealink
2. Enter a URL in a browser, that is in the same LAN subnet as the phones you wish to upgrade and use this example to build the URL: https://yourFQDN:5001/provisioning/UniqueID/firmware/yealink/NameOfROMfile.rom
3. If the file downloads successfully then you know the ROM file is accessible to the phones, and since provisioning uses the same ports then I guess provisioning should also work if this test is successful.

 

[InSite TJW]

Thanks for your advice @JohnS_3CX - I assume the path you are suggesting that I check in step 1 is a path on the PBX server. Since that is hosted for us in the cloud we don't have access to the OS it is running on.

However I was able to find a unique ID in the 3CX management console from the 'Provisioning Link' given in the 'Phone Provisioning' tab of an extension's settings. I used this along with the names of the firmware files I had downloaded from 3CX for manual firmware updates to construct the following URLs:

• https://ourFQDN:5001/provisioning/someIDthingy/firmware/fanvil/x3s_2.10.0.6646.z
• https://ourFQDN:5001/provisioning/someIDthingy/firmware/yealink/CP920-78.84.0.35.rom

Both seemed to work when I tried downloading to my PC, both on the normal data subnet and when I moved my PC to the phone subnet. I also successfully pinged ourFQDN from two of the Fanvil phones (one on the new firmware and one on the old). I manually upgraded one of the Yealink CP920s using a ROM file with the name listed about and was able to ping ourFQDN from that with 0% packet loss. However when I try to provision (they do show up on the Phones list as New) after clicking 'OK' on the 3CX Management Console it just sits there with "No Service" in the top-left corner of the screen.

One thing we have found is that the router does not like NTP (Network Time Protocol). We have managed to get NTP working on other devices by getting an NTP server configured on the router and using that (attempts to use external NTP servers still fail). Are there any other protocols used by 3CX which might be 'disliked' by the router? I also have grounds to suspect that there may be NAT issues; would 3CX be effected by NAT at all or does the SBC take care of any possible issues?

 

[JohnS_3CX]

The SBC will take care of the SIP and SDP components, but the provisioning is done over the internet via your PBX-defined HTTPS port (usually 5001). It helps you traverse NAT when it comes to calls by running this traffic through its tunnel, since NAT traversal problems would affect your calls.

Once you assign the devices via the MC, The SBC will notify the phones where to look for provisioning but once they are told, they need to access the internet independently and fetch the provisioning configuration file, as well as the firmware from the PBX sitting in the cloud. Since you were able to download the ROM from the phones subnet then there is a very good chance they are reachable to the phones too.

Try the following though in your MC: Settings > Security > SSL Transport and Ciphers - disable

Often devices that do not support TLS 1.2 will not be able to provision, and lowering the security of SSL might allow you to provision them. It might also be the case that the router interferes with TLS unexpectedly and not be the phone's fault

 

[InSite TJW]

Thanks again @JohnS_3CX for replying to my queries. I have now tried clearing the "SSL Transport and Ciphers" checkbox and this does not appear to have made a difference to the suituation.

Does 3CX use UDP or TCP for provisioning / firmware updates? During preperations for the move to our new premises, I noticed connection issues when trying to connect to a server in the old building using UDP from the new building's network. I am unable to confirm, but I suspect that issue was NAT related (we worked around it at the time by using a 4G mobile network and haven't fixed or identified the underlying issue) and only concerned UDP connections. Is it plausible that NAT issues with UDP would effect phone provisioning?

 

[Lee Cramman]

As far as I can gather from 3CX docs, for SBC UDP is required.

Given your IP address will have changed, have you checked your cloud PBX firewall accepts incoming TCP/UDP connections on port 5090 from your new IP (and, for that matter, has the appropriate ports open to your new IP in order to allow provisioning)? A port scanner may help you with this.

My experience with firmware is that you can often, but not always, get away without doing it the manual way. Generally, I try a provision / update first and, if that fails, fall back to updating it manually. That's not exactly 'by the book' though...

 

[cobaltit]

SBC or not both are potentially needed. Notifications are sent via SIP which is UDP by default, and then the actual download of firmware or configuration files is done via TCP

 

[InSite TJW]

Thanks for the replies. @Lee Cramman how do I check whether this could be a firewall issue? Everything seems to be coming up green on the Firewall Check tool in the management console.

 

[BrenttG]

did you re-ip the SBC, and if so, did you reinstall it and change its IP to the new scheme? its not as simple as just changing the IP of the box its running on, its easiest if you just reinstall the sbc fresh if its windows.

 

[InSite TJW]

@BrenttG our SBC runs in a Linux VM under Hyper-V on Windows 10. We have changed the private IPs of the phone VLAN, including on the SBC (if I recall correctly the SBC doesn't need a public IP; it works through NAT). To re-ip the SBC there were two text files I changed, one had the IP address and the other (possibly) the DNS server records. I think this is equivalent to changing to changing the TCP/IPv4 properties on the network adapter in Windows. Is there anything else I should have done?

 

[Lee Cramman]

Thanks for the replies. @Lee Cramman how do I check whether this could be a firewall issue? Everything seems to be coming up green on the Firewall Check tool in the management console.

It depends on your cloud provider. We use Google Compute Engine and I would just click into the NIC settings of my instance and have a look at the Network Analysis at the bottom of the page.

Our firewall has the necessary 3CX ports open but only to our (fixed) IP address so that even if someone managed to steal our credentials, they would not be able to use them from outside of our network.

I have had random problems re-IP'ing SBC's before so, given it's such a quick install, I generally blow away the SBC and re-install from scratch if there are any significant network changes. That said, your problems sound like they are related to provisioning, not SBC so that may well be a red herring.

Of course, when re-provisioning the phones you will need to make sure you use the new SBC local IP address. Not doing this and continuing to use the old IP is a pretty easy mistake to make.

Can I ask if your newly factory reset phones are appearing on the phones screen of the admin console and if you are provisioning via this method?

 

[InSite TJW]

Of course, when re-provisioning the phones you will need to make sure you use the new SBC local IP address. Not doing this and continuing to use the old IP is a pretty easy mistake to make.

Can I ask if your newly factory reset phones are appearing on the phones screen of the admin console and if you are provisioning via this method?

The provisioning method I am trying to use is selecting a phone on the Phones screen and clicking the 'Assign Ext' button, so the SBC IP address should be correct. I can see the Fanvil phones and Yealink conference phones listed as 'new' on the Phones screen, even the one that I have left on the Fanvil factory firmware (2.4.0.5487) for testing purposes (the other Fanvils I have manually updated so that I could provision them, the Yealinks can't be provisioned at all).

 

[Lee Cramman]

I seem to remember that you still need to complete the SBC IP address but that could just be my memory playing tricks on me - I'll test a Yealink phone and let you know.

edit - just did that and it picked up the correct SBC automatically just like you said.

Out of interest, try logging into one of the phones' web portals and disabling the requirement for SSL and see if that makes a difference for provisioning.

 

[Lee Cramman]

To add to my previous post, the SBC IP is automatic IF you directly assign the phone to an extension. Hot desking is a different matter and the SBC and provisioning method must be entered manually in the hot desking section before assigning the phone..

 

[BrenttG]

I think at this point, if it still isnt working, you may need to reinstall the SBC to get the config 100% solid on it, and then re-provision the phones ONLY AFTER making sure their settings on the 3CX server point correctly to the new SBC. Its just probably the quickest route to fixing it.

 

[InSite TJW]

Out of interest, try logging into one of the phones' web portals and disabling the requirement for SSL and see if that makes a difference for provisioning.

Thanks for the advice. Unfortunately, having spent a good few minutes looking (and using CTRL+F for ssl and tls in the Fanvil manual) I have been unable to find that option.

Yesterday I noticed that 3CX has a packet capture option after a colleague pointed out the Activity Log page on the Management Console. I therefore tried running a Capture while trying to provision a phone in order to test my NAT issue theory. I wasn't able to confirm that, and am unclear what each packet is doing, but I have noticed an ICMP type 3 code 13 message in there (destination unreachable (communication administratively filtered)). Not sure if that is significant, the source IP of the ICMP message is the public IP of our network and the destination is our PBX.

Given your IP address will have changed, have you checked your cloud PBX firewall accepts incoming TCP/UDP connections on port 5090 from your new IP (and, for that matter, has the appropriate ports open to your new IP in order to allow provisioning)?

I wasn't responsible for setting it up (I'm not sure, but I think our reseller did it) but I've now found out that the cloud server is on OVH. A quick Google found their instructions on configuring the firewall, but in our case it gives 'Create Firewall' instead of 'Configure the firewall' which worries me slightly as it seems to suggest that whoever setup the server for us left it without a firewall.

Another thing that happened yesterday was that we decided to try out the 3CX web meeting feature for the first time. Without actually having any attendees join, after a few seconds a 'Reconnecting Media...' message appears and does not go away. We then tried connecting a laptop to the internet via a 3G/4G mobile phone (thus avoiding our LAN) and didn't get the 'Reconnecting Media...' message.

 

[Lee Cramman]

Unfortunately, having spent a good few minutes looking (and using CTRL+F for ssl and tls in the Fanvil manual) I have been unable to find that option.

I must admit I'm rapidly running out of ideas!

I didn't explain myself well before. The setting I mentioned is for the Yealink phones, not the Fanvil. I missed that you're also having issues with newest Fanvils which makes checking that out a little pointless.

However, it would be interesting to know if you are able to factory reset, delete and re-provision a known good Fanvil phone. I have a theory that you won't be able to re-provision once the phone has been removed / reset.

 

[InSite TJW]

However, it would be interesting to know if you are able to factory reset, delete and re-provision a known good Fanvil phone. I have a theory that you won't be able to re-provision once the phone has been removed / reset.

I can do that no trouble. It's not that there are good/bad Fanvil phones, it just depends on the firmware version. The old firmware they come with from the factory used to be fine (for provisioning at least) in the old building but now they won't provision until I've manually updated the firmware. I used to process new phones by provisioning first then selecting them all in the 3CX console and hitting the firmware update button to do the whole batch at once. Now, I have to manually update each Fanvil phone's firmware before I can provision them, which is a fair bit slower.

Thank you for getting me to check I could still reset and provision the Fanvils though, I ran the packet capture this time and saw that seemed to do the provisioning using only TCP (with the old firmware that doesn't work there were some UDP packets). That probably means it is a UDP issue, so I'll have another look at the router configuration and see if I can figure out why UDP doesn't seem to be getting through.

 

[Lee Cramman]

I provision exactly the same way (even though I'm not supposed to) in order to make firmware updates on brand new phones easier. I guess if we're colouring outside of the lines then we can't really expect support from 3CX.

I don't know if you're aware or not but for Fanvil phones in post mode you can telnet in and point them to an FTP server as an alternative method of getting the firmware onto the correct version.

There is also FDMS which is a web GUI tool that enables you to bulk deploy firrmware updates (according to the blurb, I've never used it myself). I believe it's free.

I tried to check if Yealink have a similar tool but our internet connection isn't working well today and their website support website is pretty impenetrable so I had to give up.

 

[InSite TJW]

The SBC will take care of the SIP and SDP components, but the provisioning is done over the internet via your PBX-defined HTTPS port (usually 5001). It helps you traverse NAT when it comes to calls by running this traffic through its tunnel, since NAT traversal problems would affect your calls.

So it's plausible we could have NAT issues with provisioning that doesn't effect calls thanks to the SBC. Unfortunately I don't understand the packet captures well enough to identify whether NAT is the issue.

I think at this point, if it still isnt working, you may need to reinstall the SBC to get the config 100% solid on it, and then re-provision the phones ONLY AFTER making sure their settings on the 3CX server point correctly to the new SBC. Its just probably the quickest route to fixing it.

I've actually decided that I should change the IP again. In the move I selected a 172.0.Y.X 255.255.255.0 range of IP addresses for the VOIP network but have now realised that is not an official IANA private address range. At the time I must have naively assumed that all 172.x.x.x addresses were suitable. I have now asked for another VLAN to be set up with 172.20.X.X addresses (subnet mask is still 255.255.255.0) and installed a fresh SBC on the new VLAN. However, the new SBC doesn't seem to be working at all. I even temporarily shut down the working one in case the PBX was confused by two SBCs sharing the same public IP address (both being behind our router's NAT of course) but still the test phone connected to the new VLAN would not show up on the 3CX management console.

Perhaps I made a typo somewhere while installing the new SBC, is there a command to clear the SBC installation so I can reinstall the 3CX SBC software again without reinstalling Debian too? I did notice however when installing the SBC that the instructions suggest you need to 'Add SBC' on the SIP Trunks page in the Management Console. I don't see the 'Add SBC' button there, and we can't have had it before because the existing SBC isn't listed. I don't think the SBC installer asked for an Authentication Key ID either (although it did ask for a tunnel password from the management console which might serve the same purpose).