v15.5 Provisioning Issues After Server Failure

[[email protected]]

Last weekend during scheduled Windows Updates I had my Windows Server 2016 VM fail to boot. My images of the server also had the same failure and I was forced to rebuild the server from scratch. After importing the previous nights backup config everything but provisioning worked.

I found that our old install had a different file path because it was originally a much older version of 3CX that was upgraded an unknown number of times. In the new version my file path is C:\ProgramData\3CX\Data\Http\wwwroot\provisioning and I was able to recreate the new provisioning folder and copy the old data into it. I updated the system parameter for PROVISIONING_FOLDER_ROOT to the new location and when I try to browse to that URL I can see that it exists but can't view the directory contents.

Even though I have the URL working now it does not generate a config for new phones. I can see the phone show up in 3CX. I can assign it an extension but it won't configure. If I try to manually provision the phone it doesn't update anything. If I manually paste my http://provisioninglink/mac.cfg it won't give me a file to download. What I did notice is that if I use an old phone MAC that was configured years ago the old .cfg file still exists in the provisioning directory so it will find it just fine. My issue is generating new configs.

While reading through other posts I noticed that some people were using the firewall tester. When I test it tells me the below information. The firewall on this server is off and the phones talk directly to it through the local network. There is no special configurations and I tested with both 80 and 443. Both load fine in a web browser.

FAIL: Port forwarding is not correctly configured, or something in between this tool and 3CX Phone System is blocking traffic.
Possible causes:
1) Port forwarding not configured. Port forward the Web Port (default 5000 for HTTP and 5001 for HTTPS for standard installations. 3CX Cloud server installations work on port 80 HTTP and port 443 HTTPS.
2) Antivirus installed on 3CX Phone System Server blocking 3CX Presence packets
3) Windows firewall can also be blocking packets.

Debug information: The remote server returned an error: (404) Not Found.

Does anyone have any idea where to look for generating the files? Thank you in advance.

 

[innotip]

Hi,
Is there a reason for using Windows ? I would simply make a fresh debian 9 install and restore configuration from the backup.

If you really want / need to stick on that windows install, I would avoid touching any advanced parameter like this provisionning folder, why do you need to do so ?

 

[[email protected]]

We standardize on Windows in our shop. This was something that was working before in Windows until the restore and I believe it can be resolved. The link to the provisioning folder exists and is accessible via http and https. I'm not sure what generates the config files and how to troubleshoot it.

I had to move the provisioning folder because it ended up in a bad location after the reinstall. Pointing it to a new location in parameters and creating a provisioning folder in NGINX is easy to revert if needed.

 

[[email protected]]

During testing I was able to generate a config from the 3CX management console, save it to the provisioning folder, and have the phone load it.

This is where things get confusing. The correct extension loaded but it loaded with old user data. The user it loaded was removed a little while back and the new user of this extension is replacing him.

I will need to reboot this 3CX server this saturday to install some IBM AS400 software. If I can't get this figured out I'll do more testing then during the maintenance windows after I've rebooted. In the past I've had to reboot 3CX after making parameter changes and I wonder if I need to reboot after updating the location of the provisioning folder.

 

[CentrexJ]

You shouldn't muck with the parameters. What I would do is take a fresh backup, remove 15.5 which is obsolete and restore the backup to v16. Best to troubleshoot from the current version anyway.

 

[[email protected]]

If I update to v16 then I also have to update the soft client on several hundred computers. Our IT staff is at 40% due to layoffs. This is not something we're doing right now. It worked before the crash and there's no reason I can't get it working until we're prepared to upgrade.

I have no problems setting the system paramater back and I'm confused as to why people on a tech forum are so against making config changes. If I adjust the config back to the old folder as suggested the issue still remains; How do I get the web server to see that old path and get it to generate config files? With the parameter updated and the folder copied into the new web server root everything is there but it doesn't auto generate configs. What portions of the system do I troubleshoot for generating configs?

 

[CentrexJ]

The v15.5 soft phone will connect and work with v16. It will prompt them to update to the new client and they "should" " say yes and update but if they say no or ignore it they will still function.

You can see here from a v16 system there are 3cx clients with v15 and v16 working on it. In fact this client still has users using the v15 mac client because it has the switchboard which was removed from the v16 mac client.

 

[[email protected]]

That's good information to know but I'm not upgrading right now. I was trying to keep it simple with my response but we also have to take into consideration old phones that fall out of support, standardizing on phone firmware, updating all of the old templates and any other unknown issues that come up during testing before rollout. Now is not the time to force an upgrade without proper preperation.

With that said. I've been doing some reading on and off today and I still can't figure out how the system generates the configs and how I would go about testing and managing it. I'm not sure what version of 3CX we started on but I know that the original IIS paths carried over when it moved NGINX. If I migrate them in the config, move the files, and can reach them via a web browser then it brings me back to the original question. What generates the configs, how do I configure and test it?

 

[cobaltit]

I have to go back and dig to see exactly where but at some point 3CX started generating configs on the fly and no longer served them from the file system. I'd try removing the config file you placed in the file system (the one you said had old data) and then try to download/provision again. I would expect you to get the config as you see it in the management console.

 

[[email protected]]

I've got this all figured out and I want to make sure I leave some notes for people having trouble in the future. This ended up being a failure of multiple things and my troubleshooting just made it more confusing.

First off. The system is called the presence and provisioning system. I found the most info while searching for that phrase.

Next item is the firewall documentation is what led me to all the 3CX info that helps troubleshoot this. Note that I didn't have to make any changes to the firewall, just that there's a lot of good information about 3CX and networking in there.
https://www.3cx.com/docs/firewall-checker/
https://www.3cx.com/support/firewall-configuration/
https://www.3cx.com/docs/manual/firewall-router-configuration/#h.2b54zvy76urs

I tested 4 URL's to verify what parts of provisioning were working. 2 for FQDN and 2 for local ip address.
http://fqdn/provisioning/randomfolder/mac.cfg
https://fqdn/provisioning/randomfolder/mac.cfg
http://ip/provisioning/randomfolder/mac.cfg
https://ip/provisioning/randomfolder/mac.cfg

What I found is that the provisioning system was only responding to fqdn requests. If I tried to reach the provisioning server by ip it would not generate a config. When I tried to switch my option66 in DHCP to issue the FQDN it was resolving to IP before it got to the client resulting in the provisioning not working. But if I manually log into the yealink phone and point it at the fqdn of the provisioning server it would work just fine.

I had been doing a lot of troubleshooting at this point when I realized the listener for http/https was only responding to fqdn. So I had to roll all of my changes back to the original configs. I also went through the system parameters and searched for local ip, fqdn, and the word PROVISIONING. I verified all paths where correct, permissions were correct, and that the fqdn resolved properly in DNS. This is what I came up with from going over everything.

1. The provisioning directory should not be in the http directory. It will try to treat it like a standard web directory if you do this. Make sure the provisioning path is not in the http directory for NGINX.

2. There's no need to edit the nginx.conf file. During troubleshooting I thought the listener command in nginx.conf needed updated and it was unnecessary.

3. Old versions of 3cx (prior to 12) used port 5000 and 5001. On the crashed server we had 80, 443, 5000, and 5001 working. IIS had also been installed on the crashed server. I believe the first part of my problem came from losing port 5000 and 5001 on the new server. I had to make sure that DHCP option 66 had the correct port and that the parameters in 3CX had port 5000 removed from them.

4. I had a single parameter in 3CX that had a capital P in provisioning. Make sure your configs and folder paths are the correct case.

5. There are very few configurations for managing provisioning. If you think something in provisioning simply isn't working reinstalling is your best option to troubleshoot.

By the time I got done with the above I had it working again. It responds on both http/https for fqdn and ip.