Security Certificate not renewing

[gregsmith0]

This is an ongoing problem. please also see https://www.3cx.com/community/threads/3cx-security-certificate-is-not-renewing.80754/

We have been manually renewing the certificates with lets encrypt. This time when we go to renew our certificate lets encrypt tells us that we can't update as .3cx.co.uk has too many certificate requests.

I'll accept a solution to this part of the problem but would prefer a full solution to the ongoing problem of the service not automatically updating the certificate.

 

[VasilisV_3CX]

Hi @gregsmith0,

Why are you renewing the certificate yourself manually? How many requests have you sent for renewal over the past 7 - 10 days?

 

[gregsmith0]

Hi Vasilis

We are renewing manually because if we leave it to auto-renew, we get no email messages telling us that is failing and then it fails leaving us with 120 odd users who cannot connect via the WebClient, which is now by far our most popular method of users accessing the phone system.

I used to get emails telling me that the SSL certificate had been successfully renewed. the last time was july 2020.

I have attached the certificate log which is basically repeating the same log every 12 hours. I don't think it has anything which is a security risk but if so please let me know or delete it.

Thanks for getting to me so promptly

Greg

 

[VasilisV_3CX]

Hi @gregsmith0,

Please note that we cannot do anything to work around the 7 - 10 day wait.

We can check why you're not receiving emails after the 10 days has passed.

 

[NickD_3CX]

Hi Greg,

Yeah, the log did kind of have sensitive data, like your license key!
Please be careful what you upload.

 

[gregsmith0]

Sorry Vasilis

I don't quite understand what you are getting at with the 7 to 10-day thing or why you can't investigate. in 10 days the security certificate will have expired and I will be once again unable to access via the Web. In case it is relevant we reboot our server at 4am GMT each day. We do this because if we don't, the operating disk fills up and runs out of space over a 7 to 10 day period.

 

[Saqqara]

Why is the operating disk disk filling up , have you investigated this and which folders are involved?

What operating system are you using ?

What are your settings for log files - below is the defaults

How much disk space is assigned to the server ?

 

[gregsmith0]

Hi Saqqara

I am using those settings. I set the system up in late 2018 using the express method Debian 9 and a google cloud instance

I have a 10gb hard disk and normally about 3-4 gb is free.

We never got to the bottom of what was filling up the hard disk although a reset clears the problem so it is something fairly transitory!
Greg

 

[NickD_3CX]

Hi!

The reason there is not a lot we can do right not is because the error you said you get "lets encrypt tells us that we can't update as .3cx.co.uk has too many certificate requests." means that you have requested from LE to re-issue the certificates more than 5 times within the last 7 days.
Now basically this domain is 'blacklisted' by LE for 7 days and there is nothing we can do about it. That's the only scenario that I am aware of where LE returns this error.

The, best I can suggest is to wait it out and let the system do the renewal by itself. This though would means that you should avoid restarting the server at night, as this happens as a background task in the early hours (random time), so a restart may interrupt it. Also, don't shut the server down at all for the same reason.

If you reach 48 hours before the expiry, then try and do a manual update, but in the meantime, I would not attempt anything.

 

[gregsmith0]

Thanks Nick

We got the message from Lets Encrypt on our very first request, the text of the problem is

Obtaining a new certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for: 3cx.co.uk: see https://letsencrypt.org/docs/rate-limits/

so I doubt that that is the problem. Having read Lets Enrypt's rate-limits text my interpretation would be that there are some other people out there doing something similar to us and on this occasion, we have found that 5 of them have renewed in the last week and we are now at the back of the queue.

I have stopped the machine restarting overnight and we will see if that has an effect on the automatic allocation of a certificate. I would however point out that the server is down for 45 seconds at 4 am, I time when, as far as I can see, no cron jobs are scheduled to run. The scripts that run the automatic update seem to run at 10 am and 10 pm so unless shutting the machine down upsets some arithmetic within those scripts I am baffled by why that would affect the allocation of the certificate.

 

[NickD_3CX]

Having read Lets Enrypt's rate-limits text my interpretation would be that there are some other people out there doing something similar to us and on this occasion, we have found that 5 of them have renewed in the last week and we are now at the back of the queue.

Or that, yes, someone is requesting directly from LE to get a certificate for your domain. Is it possible you have set up some other mechanism for generating the certificate for some other usage? Maybe using Certbot for some other service on this PBX machine?

Remember, this error is coming from LE and we pass it along as is.

I would however point out that the server is down for 45 seconds at 4 am, I time when, as far as I can see, no cron jobs are scheduled to run. The scripts that run the automatic update seem to run at 10 am and 10 pm so unless shutting the machine down upsets some arithmetic within those scripts I am baffled by why that would affect the allocation of the certificate.

You won't see any cron job, the cert renewal is random time between 2am and 7am if I recall and the "timer" is in the software itself.
We have had a few cases were certs weren't being renewed because the server was being shut down during the night!