Certbot fails on private mcu setup

[illcs]

Have deployed a couple of times no issue but got this today any ideas:

Adding 3CX repository... done
Updating packages... done
Installing certbot... done
Creating configuration files... done
Configuring Certbot update... done
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See t he logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for mo re details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See t he logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for mo re details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See t he logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for mo re details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See t he logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for mo re details.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 496, i n _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_e ffort)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^
File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2024-10-28 22:16:38,700:ERROR:certbot._internal.log:Some challenges have failed.
CertBot Error. Read above and fix problems.

 

[LeonidasG_3CX]

Hi,

The best thing to do here is to read the log under /var/log/letsencrypt/letsencrypt.log and see what it's complaining about. The most common reason for certbot failing for cert creation/renewal is port 80 closed on the MCU which is what it uses for the renewal/creation of the cert.

At this point i would start a fresh VM, resolve errors in the MCU log,
ensure port 80 is opened and try again.

 

[illcs]

@LeonidasG_3CX this was a working mcu until we tried to use it for a meeting yesterday so, we know firewall rules are ok but we did double check them.

Had to switch back to 3CX mcu for the meeting glad that was quick and easy.

Used this to troubleshoot https://www.3cx.com/docs/mcu-troubleshooting/ this command yield nothing Systemctl status | grep mcu, yet it was a working machine.

We next reinstalled the vm twice with the same results each time failing with certbot. Tried again this am same results. Attached the letsencrypt log

 

[LeonidasG_3CX]

I've gone ahead and removed the log file from the post. To communicate log files i suggest sending a PM next time.

I checked on our side and it seems:

1) Yesterday during the night hours, a request from your PBX to delete your FQDN arrived on our side likely from the PBX Admin UI deleting the MCU and so the old MCU DNS entry on our side was deleted.

2) We see further attempts to recreate the same DNS record likely either from attempts on your side to use the same install script/install command used for the old deleted MCU which is the wrong procedure here.

The way to correctly install a new MCU is to create a new MCU from your PBX UI, get the new install command, make a new fresh clean VM and install it there to ensure success.

 

[illcs]

The mcu had already been deleted so that dns record in that log file should have been deleted already?
Let me clarify the timeline:
Discovered private mcu was not working, went to gui it was red clicked the 3 dot menu and realized all we could do is delete so we located the troubleshooting document referenced above, ssh to private mcu, none of the steps resulted in the mcu coming back online. We need a working mcu for a meeting, so we deleted the private mcu and went to 3CX mcu. Later after the meeting we tried to create a new private mcu which is where got the error we, deleted and recreated the mcu 3 times and reinstalled a fresh vm 3 times verifying firewall rules even deleting and recreating port 80. We suspect that reason the private mcu went offline is that the certbot renewal failed at some point and is the reason we can't install a new one.

 

[MarcelloV]

Hello @illcs , your MCU FQDN was deleted from PBX UI on 2024-10-28 21:38:50 UTC. The log you provided is related to this MCU: t0k42
There is no FQDN entry for this MCU so Let's Encrypt certbot will surely fail. Do not reuse old scripts after you deleted the instance from UI.

We don't see any active OnBoard MCU record associated to your PBX right now, you can create 2 OnBoard MCU servers on your PBX.

You can start by creating a new one, installing it copying the install script from UI and see if SSL certificate is generated correctly. If not, send me a PM and I'll help you troubleshoot.

 

[illcs]

Thats, correct we have no active private mcu awaiting your input.

We will try again, but for the record each time you create a mcu there is a popup window with a curl script each time we did this we copied and pasted that script into ssh each time the same result.

 

[MarcelloV]

Ok. If the newly installed onboard MCU is not working, leave it configured (do not delete VM and do not delete instance from UI) and switch to cloud webmeeting. This way you can use webmeeting normally while we troubleshoot your issue.

 

[illcs]

and just like that failed again will pm you.
Adding 3CX repository... done
Updating packages... done
Installing certbot... done
Creating configuration files... done
Configuring Certbot update... done
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
CertBot challenge failed, retrying in 10 seconds
Requesting SSL certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2024-10-29 15:01:49,291:ERROR:certbot._internal.log:Some challenges have failed.
CertBot Error. Read above and fix problems.

 

[MarcelloV]

I see. Please provide /var/log/letsencrypt/letsencrypt.log in PM so we can check.

 

[illcs]

@MarcelloV found the solution turns out os image had ufw on though no mention on host firewall page which had correct ports open. Thank you @MarcelloV

 

[MarcelloV]

Thanks for reporting back, I'm glad the problem is now fixed!

Quick recap: in this case, ufw was installed and enabled by default on the OS (which is not the standard for Debian 12) and it had to be disabled with

sudo ufw disable

OnBoard MCU requires port 80, 443 TCP opened (80 only for Let's Encrypt certificate update every 90 days) and 48000-65535 UDP range for media streams.

Let's Encrypt error log reported

Timeout during connect (likely firewall problem)
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Let's Encrypt servers could not connect to port 80 on the OnBoard MCU and the SSL certificate could not be renewed.

Disabling ufw and reinstalling the OnBoard MCU fixed the issue.