Good afternoon,
I recently found myself in a situation where I was unable to solve this issue and was needing support. I feel after my experience with it in the last 30 hours, I should give a heads up to anybody who ever needs to contact/pay for this support.
Quick over view: I started @ this company about a year ago. We have 15 instance of 3cx across the country. I am responsible for 9 of them directly. Most are on-prem, some are on azure. The one that is having the main issue, is the one at my location (also our largest volume one). It had been having issues renewing the SSL cert for some time (as in months) Now given that we have 9, we see these messages regularly. They also often resolve themselves over time. (currently 7/9 have renewed without issue, 2 have the issue still. 1 of them being the one in question)
We started to have issues with this phone system directly a few days ago. Now, as this is only part of a very large job I manage I don't go into the management portal more than once or twice a month. So to my surprise when I went to login ... it would hang after i put my user-name and password in. After a minute it would come back with a 504 error. So I googled it, and visited this forum and most people advised to un-install and restore from backup. This was my first shock. Your first trouble shooting step, should not be to un-install an entire phone platform and re-install it. That's absurd. I would use myself as an example of why that's not a good plan, but I will get to that later.
So at this point (we are using Debian version) I realize, this is going to take actual work to figure out. So I login to the box, and start look at logs, and running service and such! I take a snapshot of the system (Vmware) and proceed to look around. I start and stop the management service, reboot the server, follow guides from the Forums. Nothing works, the issue is the same. Now another issue comes up, and i have to put this down for a week. When I get back in the office, I get notified that several of our numbers are now ringing the fail line instead of going into 3cx. So I try to log into the Management portal, and I am now seeing a HSTS blocking. The letsencrypt cert expired on the 7th of February. So I think, ok there must be a way to renew the certificate and after some googling i find it! PbxConfigTool -renew-certificate i run it. Nothing happens. So i start digging into the logs... I find the following error.
1544#1544: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
Oh man, I can not find anything on how to solve this. I google this, and it appears to be a rate-limiting thing imposed by letsencrypt. I try to follow some documents from letsencrypt themselves and nope.. im not blocked, im not rate limited (on my side). So I dig a litter deeper, and we are using the 3cx FQDN service (aka they handle the SSL cert for us) and that means they may have the issue. So I reach out first to our Reseller. They say, nothing we can do to help. But they suggest we remove and re-add 3cx. At this point, my phone system is now not working, so i figure whats the worse that can happen. I take the day priors backup, and generate a new backup (via CLI) and store them. I add another snap shot of the VM. I remove (apt remove 3cxpbx) and re-add it. Now it comes down with v16 of the software. I boot up the management portal...and boom. Turns out the certificate is part of what is backed up.
So now I am stuck with trying to figure out how to force renewing it. I check through the PbxConfigTool.log file and it sees that its a letsencrypt and all that but stops renewing it. So I think, ok lets remove the cert and try renewing. That doesn't fix it. So i restore the cert, and roll backup the server to the initial snapshot. My phone system is still down, but its back to its original state. I realize, ok I will need help.
So I purchase the $100USD support, create the ticket, explaining the above. Here is the snippet I enter on the ticket:
"
For some time the letsencrypt service has been failing. Normally, this happens and time resolves this. We have now lost access to the management entirely. It gives HSTS error as the cert expired Feb 7th. When i got into console, and try PbxToolConfig -renew-certificate it failes, the log shows:
2020/02/12 13:38:48 [error] 1544#1544: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
I currently can not get into the management console at all, i can not check/upgrade or backup anything.
How do i fix this ASAP.
"
Then begins the worst Tech support I have ever received.
[4hrs later]
He first suggests that I login to the server and run pbxtoolconfig -renew-certifcate
He suggests I log into the server and verify that nginx is running.
I reply by stating; you should re-read what I sent. I already did that, and nginx is obviously working as I am seeing a HSTS error.
[2hrs later]
At this point, i assume he will realize that basically what needs to happen is for my certificate to be renewed and the (6: unauthorized) error is the source of where to start...[Also i have to point out that he was cutting and pasting from canned response he uses since it was labeled and numbered and he just cut out certain steps] He then asked for my nginx logs and nginx config.
[1 hr later]
Which i send him, and he comes back stating [You changed the nginx config and removed http2, thats the problem]
So i change it back (obviously not the issue) and guess what. Not fixed.
[1hr later] I ask my ticket to be escalated to Tier 2 or another ticket. I didnt pay 100USD for a google search support.
[1hr later]
He then asked for remote access to SSH into the system. I check to see what accounts are on the system i see a pbxsystem account, so i make the mistake of assume is a built in service account. He gives me his IP and i make the firewall rule. I then move to one of my remote locations, and test the SSH. It works. Anydesk to my home PC and test. it works.
[17hrs later]
I get a message that i didn't supply him the root, password. He also claims he could not connect. So i reply, and give him both, and verify that its working (from multiple sites).
[2hrs later] Nothing. So I ping again that I wish to have my ticket escalated.
[2hr later] I request my ticket is reviewed by the Support Supervisor. Who comes back and says, there is nothing wrong as there is a 48hr SLA. He also states, I'm not giving the support agent access. I reply by stating and including screen shots of it working remotely with those credentials. I also include a screen shot of my firewall rule.
I'm nearing day 2 of this, and so far I have paid 1$00USD for a google search and a person faking a lack of connectivity and have had ZERO change or value added by contacting them.
It's a real shame. 3cx the product is great when it works. However, now I have to re-envision my entire national strategy as I can't rely on contacting technical support for something as simple as renewing a SSL cert. Could you imagine I was having an real issue? I am actually glad this happened now, as I was about to consolidate those 15 3cx instances into larger multi point azure instances.
Just a heads up to other IT folks out there. 3CX Product is good, but their tech support is absolutely abysmal and you have to pay per ticket. Well, you get them googling for you for 100USD.
-Andrew
I recently found myself in a situation where I was unable to solve this issue and was needing support. I feel after my experience with it in the last 30 hours, I should give a heads up to anybody who ever needs to contact/pay for this support.
Quick over view: I started @ this company about a year ago. We have 15 instance of 3cx across the country. I am responsible for 9 of them directly. Most are on-prem, some are on azure. The one that is having the main issue, is the one at my location (also our largest volume one). It had been having issues renewing the SSL cert for some time (as in months) Now given that we have 9, we see these messages regularly. They also often resolve themselves over time. (currently 7/9 have renewed without issue, 2 have the issue still. 1 of them being the one in question)
We started to have issues with this phone system directly a few days ago. Now, as this is only part of a very large job I manage I don't go into the management portal more than once or twice a month. So to my surprise when I went to login ... it would hang after i put my user-name and password in. After a minute it would come back with a 504 error. So I googled it, and visited this forum and most people advised to un-install and restore from backup. This was my first shock. Your first trouble shooting step, should not be to un-install an entire phone platform and re-install it. That's absurd. I would use myself as an example of why that's not a good plan, but I will get to that later.
So at this point (we are using Debian version) I realize, this is going to take actual work to figure out. So I login to the box, and start look at logs, and running service and such! I take a snapshot of the system (Vmware) and proceed to look around. I start and stop the management service, reboot the server, follow guides from the Forums. Nothing works, the issue is the same. Now another issue comes up, and i have to put this down for a week. When I get back in the office, I get notified that several of our numbers are now ringing the fail line instead of going into 3cx. So I try to log into the Management portal, and I am now seeing a HSTS blocking. The letsencrypt cert expired on the 7th of February. So I think, ok there must be a way to renew the certificate and after some googling i find it! PbxConfigTool -renew-certificate i run it. Nothing happens. So i start digging into the logs... I find the following error.
1544#1544: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
Oh man, I can not find anything on how to solve this. I google this, and it appears to be a rate-limiting thing imposed by letsencrypt. I try to follow some documents from letsencrypt themselves and nope.. im not blocked, im not rate limited (on my side). So I dig a litter deeper, and we are using the 3cx FQDN service (aka they handle the SSL cert for us) and that means they may have the issue. So I reach out first to our Reseller. They say, nothing we can do to help. But they suggest we remove and re-add 3cx. At this point, my phone system is now not working, so i figure whats the worse that can happen. I take the day priors backup, and generate a new backup (via CLI) and store them. I add another snap shot of the VM. I remove (apt remove 3cxpbx) and re-add it. Now it comes down with v16 of the software. I boot up the management portal...and boom. Turns out the certificate is part of what is backed up.
So now I am stuck with trying to figure out how to force renewing it. I check through the PbxConfigTool.log file and it sees that its a letsencrypt and all that but stops renewing it. So I think, ok lets remove the cert and try renewing. That doesn't fix it. So i restore the cert, and roll backup the server to the initial snapshot. My phone system is still down, but its back to its original state. I realize, ok I will need help.
So I purchase the $100USD support, create the ticket, explaining the above. Here is the snippet I enter on the ticket:
"
For some time the letsencrypt service has been failing. Normally, this happens and time resolves this. We have now lost access to the management entirely. It gives HSTS error as the cert expired Feb 7th. When i got into console, and try PbxToolConfig -renew-certificate it failes, the log shows:
2020/02/12 13:38:48 [error] 1544#1544: OCSP response not successful (6: unauthorized) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
I currently can not get into the management console at all, i can not check/upgrade or backup anything.
How do i fix this ASAP.
"
Then begins the worst Tech support I have ever received.
[4hrs later]
He first suggests that I login to the server and run pbxtoolconfig -renew-certifcate
He suggests I log into the server and verify that nginx is running.
I reply by stating; you should re-read what I sent. I already did that, and nginx is obviously working as I am seeing a HSTS error.
[2hrs later]
At this point, i assume he will realize that basically what needs to happen is for my certificate to be renewed and the (6: unauthorized) error is the source of where to start...[Also i have to point out that he was cutting and pasting from canned response he uses since it was labeled and numbered and he just cut out certain steps] He then asked for my nginx logs and nginx config.
[1 hr later]
Which i send him, and he comes back stating [You changed the nginx config and removed http2, thats the problem]
So i change it back (obviously not the issue) and guess what. Not fixed.
[1hr later] I ask my ticket to be escalated to Tier 2 or another ticket. I didnt pay 100USD for a google search support.
[1hr later]
He then asked for remote access to SSH into the system. I check to see what accounts are on the system i see a pbxsystem account, so i make the mistake of assume is a built in service account. He gives me his IP and i make the firewall rule. I then move to one of my remote locations, and test the SSH. It works. Anydesk to my home PC and test. it works.
[17hrs later]
I get a message that i didn't supply him the root, password. He also claims he could not connect. So i reply, and give him both, and verify that its working (from multiple sites).
[2hrs later] Nothing. So I ping again that I wish to have my ticket escalated.
[2hr later] I request my ticket is reviewed by the Support Supervisor. Who comes back and says, there is nothing wrong as there is a 48hr SLA. He also states, I'm not giving the support agent access. I reply by stating and including screen shots of it working remotely with those credentials. I also include a screen shot of my firewall rule.
I'm nearing day 2 of this, and so far I have paid 1$00USD for a google search and a person faking a lack of connectivity and have had ZERO change or value added by contacting them.
It's a real shame. 3cx the product is great when it works. However, now I have to re-envision my entire national strategy as I can't rely on contacting technical support for something as simple as renewing a SSL cert. Could you imagine I was having an real issue? I am actually glad this happened now, as I was about to consolidate those 15 3cx instances into larger multi point azure instances.
Just a heads up to other IT folks out there. 3CX Product is good, but their tech support is absolutely abysmal and you have to pay per ticket. Well, you get them googling for you for 100USD.
-Andrew
