Unable to force Let's Encrypt SSL renewal

[Felicia King]

We have a client who refused to go on proactive management on the PBX. As a result, when some items changed in how 3CX worked in a v16 update their Let's Encrypt SSL cert was failing to renew. I got an alert about it and told them that no action meant that there was going to be a problem with the PBX. They finally 90 days later sent us some funds to work on it.
I've corrected the issue that was preventing the PBX from being able to successfully renew the cert from a communications perspective. However, the cert is still not renewing. I reboot the PBX and that is not enough to cause it to try to get a new cert. I can confirm that the network communications are successfully going to activation.3cx.com.
I tried launching PbxConfigTool -renew-certificates and that does not cause the cert to renew either.
As far as I can tell, the only way it will renew is to wait until 4A tomorrow. Seems like there should be a way to force the PBX to try to do the renewal now, but I cannot find the procedure to make that work.

Does anyone know how to force the certificate renewal process on Debian?

 

[innotip]

Hi, is your firewall checker pass ?

 

[Felicia King]

The firewall checker is not the problem. I have seen this same exact thing with four other 3CX PBX. What I observed previously was that the only way to know if the SSL cert was going to renew successfully or not was to wait until the 4A process.
I would like to get this PBX working now with a valid cert now, not wait until 4A tomorrow morning. The problem that was causing the PBX to not renew the cert is resolved. I'm asking how to force the PBX to actually do the cert renewal.
The PBX does not retry every few hours. I've seen this on several PBX. It only attempts to renew the cert once a day at 4A.
The documented procedure on using PBXConfigTool to renew the certs executes successfully from the perspective that the command runs, but the renewal does not work. And I have confirmed that the network traffic is working properly and being allowed.

 

[innotip]

maybe you can try to cheat and switch to dynamic IP, wait few seconds, and then switch back to static ? Anyway there is usually no maintenance to do on the letsencrypt things if used normally

 

[Felicia King]

The CLI commands for the apps inside of /usr/lib/3cxpbx are also not documented anywhere. I've been through the manual and no CLI descriptions to be found for the apps on Debian. So not helpful.

 

[innotip]

no need for CLI with what I suggested, did you try it ?

 

[Felicia King]

It is not possible to take the PBX down or offline nor is the risk of a reconfiguration worth it when the alternative is to wait until 4A tomorrow. I just think it is really disappointing that 3CX appears to not have the CLI options documented and what I have been able to find does not work. 3CX does not make it easy for partners to support the PBX.

 

[Felicia King]

~1:30A today the PBX successfully renewed the cert.
Is there anyone from 3CX that can comment on what the procedure is to force the renewal? Waiting until the next day to find out if it is going to work or not is not a viable solution.

 

[cobaltit]

Just wanted to make sure I understand this.

- You have a customer created situation because they declined proactive maintenance and thus created this 'emergency'.
- You said you've seen this before and you seemed pretty confident you resolved the issue preventing the certificate from being renewed.

With the issue effectively being resolved why is waiting for the normal (known) renewal interval not a viable solution? You seemed to be confident the issue was resolved, and it would seem waiting for the interval (which worked) would basically just increase the customer's pain point which, while not ideal, does reinforce the importance of having the aforementioned proactive maintenance which they declined.

I guess I don't see a problem here?

 

[Felicia King]

The problem is that there is no documented way to initiate a command on the PBX in Debian that will tell the PBX to initiate a process whereby it attempts to renew its certificate.
This command does not work.
PbxConfigTool -renew-certificates
It does not error out, but it does not achieve the objective.

The problem? Having the webUI for the PBX down for >18 hours simply because there is no documented procedure to tell the PBX to renew the cert is a problem.

Furthermore, if the intent is to make a change and then get the PBX to attempt to renew the cert so you can do troubleshooting to find out if the modification fixed the problem, well there's no way to do that other than to wait until the next day.

So the issue is not just one day's worth of outage. It is many days worth of outages.

When I first saw this problem, it took several days of outages on one PBX to troubleshoot it because I had no way to get the PBX to attempt to renew the cert.

It's not viable to wait days when there should be a simple command to tell the PBX to attempt the renewal.

 

[cobaltit]

So there is a simple way to tell the PBX to renew the cert:

https://www.3cx.com/docs/fqdn-ssl-certificate-v15/

Otherwise if you go the 3CX managed FQDN, you deal with the limitations. 3CX doesn't need to document command line options that aren't intended for customers to use. Using those incorrectly can cause more problems than they solve, including going over the limit on renewal attempts. If the need was that dire you also could have taken a backup, uninstalled and then reinstalled which would have forced the SSL renewal during the restore.

 

[Felicia King]

It's interesting that 3CX strongly advocates the subscription model where they supply the certificate, DNS, FQDN registration, and it is pretty highly automated. I don't see that as limitations. It decreases the TCO for the owner of the PBX. It means less manual labor annually or every two years when the cert expires, even ignoring the cert cost.

Neither of the options you suggest are sensitive to the support budget of the customer OR bigger outages. How is acceptable to do a total down of the PBX and increasing the risk by doing a reinstall/restore unless there is no other option? If I was the customer, I would not buy that as a viable support action and I would probably look sideways at anyone who suggested that to me for my business.

Even for a company that has a wildcard cert, using a custom cert is not sensitive to TCO for the company because it is more expensive to maintain strictly from a labor perspective.

We are not customers. We are partners. We are supporting the phone systems for customers. I was contacted privately by 3CX support about the issue and provided the syntax.

Where the system falls down completely is that there is a hole in the documentation. Where is the article that says: LE rates limits exist for renewals on the 3CX provided certificate system. We recommend that the PBX certificate be allowed to renew this on its own. This process will occur once per day for systems that detect a renewal is required. This process occurs sometime between 1A and 4A. If you need to force a renewal, contact support. Certificate renewals cannot be forced by rebooting the PBX. Certificate renewal attempts communicate with activation.3cx.com. 3CX may change its geo-hosting for this resource at any time. If you run a high security network that uses geo-IP blocking, we recommend setting an exception to Geo-IP blocking for the FQDN activation.3cx.com"

There's no article that says any of that. Lack of proactive communication through product documentation is the problem. The solution is not for me to blow hours of time and create a larger risk profile for the client when the fix is some appropriate documentation. I should not have to engage in "workarounds" simply because the manufacturer of the product does not communicate the details needed to effectively support the product to its support partners. I don't think it is a big "ask" to request better documentation for the system so partners can support the systems more effectively.