Something wrong with 3CX generated SSL

[jmatano]

I could really use some help with this. As of this morning (I noticed around 9am EST), that our 3CX provided SSL link is coming up with a error. It is saying the following:

Your connection is not private​

Attackers might be trying to steal your information from (removed).nj.3cx.us (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_DATE_INVALID

When I expand Advanced:

(removed).nj.3cx.us normally uses encryption to protect your information. When Google Chrome tried to connect to -.nj.3cx.us this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be (removed).nj.3cx.us, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Google Chrome stopped the connection before any data was exchanged.

You cannot visit (removed).nj.3cx.us right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

I have already rebooted the server. I am able to access it from the internal network with the direct IP. Please assist. Thank you in advance.

 

[cobaltit]

NET::ERR_CERT_DATE_INVALID

That tells you all you need to know. Apparently you haven't been paying attention to the emails they've been sending out regarding the SSL renewal failures. So now your SSL certificate is expired.

This is most likely because either an expired license, an out of date instance, overzealous network security, or bad DNS. Fix whichever one of these is broken and it should auto renew the cert this evening

 

[jmatano]

NET::ERR_CERT_DATE_INVALID

That tells you all you need to know. Apparently you haven't been paying attention to the emails they've been sending out regarding the SSL renewal failures. So now your SSL certificate is expired.

This is most likely because either an expired license, an out of date instance, overzealous network security, or bad DNS. Fix whichever one of these is broken and it should auto renew the cert this evening

The only email or notification I received cert related from 3CX was that it in order to upgrade the cert, V16 Update 5 is required, which is where our instance is currently at. DNS for the domain is a static route, and the phone system works fine. License doesn't need to be renewed until July.

 

[cobaltit]

It wouldn't be from 3CX. It would be from your 3CX instance to the notification email address. Same place other notifications go (backup, blacklist, etc). It would be from [email protected] (if using 3CX SMTP) and it would say this if successful it would have a subject like "3CX Phone System - <Registered User/Company> - 3CX Notification: SSL Certificate Renewal - <3CX FQDN>

The SSL Certificate for your 3CX installation at 3CX FQDN was automatically renewed for the next 90 days. 
Your 3CX web server is now secure and all web traffic that passes via 3CX is encrypted

Or if it failed, it would have a subject of "3CX Phone System - <Registered User/Company> - 3CX Notifcation: SSL Certificate Renewal Failed - <3CX FQDN>" and look like this:

The SSL Certificate renewal for 3CXFQDN failed - Error:
IpUpdater.FqdnGenerationException: Error creating FQDN: LetsEncrypt failed to validate domain. Please try again in 5 minutes.
   at PostInstall.CertificateGenerator.ProcessCertificatesDirectory(String directory, Boolean temporaryCertificateGenerated, Int32 regenerateNotSelfSignedCertificatesFrom, Int32 regenerateNotSelfSignedCertificatesTo, CloudServerStatus statuses, Int32 regenerateCertificateExiredInDays, String appBin, UInt16 sipPort, UInt16 tunnelPort, Nullable`1 httpPort, Nullable`1 httpsPort, Boolean isPassiveFailoverMode)
   at PostInstall.CertificateGenerator.RenewCertificates(String appBin, String nginxConfigFolder, String configurationPath)

 A communication error occurred between the DNS Servers and LetsEncrypt. This rarely happens and is usually resolved on the second attempt. 3CX will try again.

 

[cobaltit]

The only email or notification I received cert related from 3CX was that it in order to upgrade the cert, V16 Update 5 is required, which is where our instance is currently at. DNS for the domain is a static route, and the phone system works fine. License doesn't need to be renewed until July.

Not sure what DNS for the domain is a static route means. DNS and routing tables are different things. What are you using for DNS? Google, ISP, internal?

And if you are talking about the email from 3CX regarding activation you might want to re-read that as it says you need to be on Update 6, not Update 5. This is probably the source of your issue.

 

[jmatano]

It wouldn't be from 3CX. It would be from your 3CX instance to the notification email address. Same place other notifications go (backup, blacklist, etc). It would be from [email protected] (if using 3CX SMTP) and it would say this if successful it would have a subject like "3CX Phone System - <Registered User/Company> - 3CX Notification: SSL Certificate Renewal - <3CX FQDN>

The SSL Certificate for your 3CX installation at 3CX FQDN was automatically renewed for the next 90 days.
Your 3CX web server is now secure and all web traffic that passes via 3CX is encrypted

Or if it failed, it would have a subject of "3CX Phone System - <Registered User/Company> - 3CX Notifcation: SSL Certificate Renewal Failed - <3CX FQDN>" and look like this:

The SSL Certificate renewal for 3CXFQDN failed - Error:
IpUpdater.FqdnGenerationException: Error creating FQDN: LetsEncrypt failed to validate domain. Please try again in 5 minutes.
   at PostInstall.CertificateGenerator.ProcessCertificatesDirectory(String directory, Boolean temporaryCertificateGenerated, Int32 regenerateNotSelfSignedCertificatesFrom, Int32 regenerateNotSelfSignedCertificatesTo, CloudServerStatus statuses, Int32 regenerateCertificateExiredInDays, String appBin, UInt16 sipPort, UInt16 tunnelPort, Nullable`1 httpPort, Nullable`1 httpsPort, Boolean isPassiveFailoverMode)
   at PostInstall.CertificateGenerator.RenewCertificates(String appBin, String nginxConfigFolder, String configurationPath)

A communication error occurred between the DNS Servers and LetsEncrypt. This rarely happens and is usually resolved on the second attempt. 3CX will try again.

The only message I got certificate related was when our instance renewed the last time, in the later half of 2020. I did not get any failure to register (or successful ones either) from during 2021.

 

[jmatano]

And if you are talking about the email from 3CX regarding activation you might want to re-read that as it says you need to be on Update 6, not Update 5. This is probably the source of your issue.

Please provide your source for Update 6 only. This is the latest communication I received from 3CX and it clearly states V 16 Update 5 or 6.

1. Incompatibility with SSL certificates
   3CX will be updating the SSL certificates which are required to activate the PBX, and allow secure connections between your installation and our cloud services. These certificates expire every 2 years and are due to be replaced on the 15th September. It is imperative that customers be upgraded to version 15.5 SP6a or v16 Update 5/6 for them to remain able to (re-)activate and refresh license key(s) information.

 

[cobaltit]

Hmm.. that does say Update 5. All the ones I have say Update 6, but they are also referencing older keys that were on v15.5 at the time so that might be why. I guess all of the v16 instances we managed were already updated so I only have emails about v15.5 versions.

Hi,
3CX will be updating the SSL certificates which are required to activate the PBX, and allow secure connections between your installation and our cloud services. These certificates expire every 2 years and are due to be replaced on the 15th September 2020. An update (v15.5.6a) is available which will enable you to (re-)activate and refresh license keys information. Make sure you update before 15th September (more information below).


Why settle for less? Upgrade to v16 Update 6 instead
________________________________________
V15.5 has reached end of life: As of 1st July 2020, this version is now unsupported, meaning there will no longer be updates available. By upgrading to v16 Update 6 you ensure ongoing maintenance of the product going forward. Simply switch to an Annual subscription, get one year free and unlock all the benefits that 3CX version 16 Update 6 has to offer!

I'd still recommend updating but I suppose you have your reasons for staying on Update 5. I'd focus on the network elements then, either a firewall messing with traffic, restrictive rules, and DNS.

 

[cobaltit]

For DNS, using Google DNS seems to be the recommendation. I tend to go with CloudFlare and don't have any issues. Here's one from someone using Quad9 that changed to Google:

Solved - SSL Certificate renewal failed - every day | 3CX - Software Based VoIP IP PBX / PABX

 

[jmatano]

Hmm.. that does say Update 5. All the ones I have say Update 6, but they are also referencing older keys that were on v15.5 at the time so that might be why. I guess all of the v16 instances we managed were already updated so I only have emails about v15.5 versions.



I'd still recommend updating but I suppose you have your reasons for staying on Update 5. I'd focus on the network elements then, either a firewall messing with traffic, restrictive rules, and DNS.

The system is functioning without issue. The website is reachable and active from WAN connections. The only issue with the system right now seems to be the cert not updating. We have a number of our agents working and relying on 3CX remotely. If DNS was a issue, we would see other issues.

I keep seeing references (once in this thread) in threads about the cert updating overnight. What process is done to update the cert and can it be manually invoked? If it runs on a scheduled job, I'm thinking there should be a way to manually run it.

 

[cobaltit]

You seem to be confused about the DNS. I'm not talking about the resolution of the 3CX FQDN by outside users. I'm talking about the DNS the 3CX system itself uses. What DNS is the 3CX system itself using?

As far as the process, yes it is scheduled by 3CX. No 3CX doesn't publish how to manually run it because if you have too many attempts in a period of time then you will be blocked from renewing (LE restrictions). You can try opening a ticket to see if support will give you the command to run it manually, but the first thing they will probably tell you to do is what I already did, and that is to change the DNS the system is using to Google.

 

[YiannisH_3CX]

Hi @jmatano

A long shot here but has happened in the past so it might be worth the question. Do you shut the 3CX server down during the night? If so leave it running as the renewal happens during the night hours. If the server is off then the renewal will not take place. It will also explain the lack of emails.