Solved 3CX SSL Certificate (3cx.us) expired (not renewing)

[Mark@EvoIT]

- 3CX SSL Certificate (*.3cx.us) expired today (10/26/17), looks like last renewed 7/26/17 timeframe.
- Restarted services, I can access with invalid warning via IE, everything is in check including maintenance (March 2018).

Looking at serial console output on Gcloud shows me this over and over (assuming culprit in here somewhere):

Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: --2017-10-26 19:45:42-- https://downloads.3cx.com/downloads/3cxpbxexpress/public.key
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: Resolving downloads.3cx.com (downloads.3cx.com)... 151.80.125.73, 158.69.11.10
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: Connecting to downloads.3cx.com (downloads.3cx.com)|151.80.125.73|:443... connected.
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: HTTP request sent, awaiting response... 302 Redirect
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: Location: http://www.3cx.com [following]
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: --2017-10-26 19:45:42-- http://www.3cx.com/
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: Resolving www.3cx.com (www.3cx.com)... 151.80.125.71
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: Connecting to www.3cx.com (www.3cx.com)|151.80.125.71|:80... failed: Connection refused.
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: gpg: no valid OpenPGP data found.
Oct 26 19:45:42 pbxexpress startup-script: INFO startup-script: >>>>> Cant get Public Key, retry....

Any help or ideas greatly appreciated, other 3x PBX's in Gcloud no issues identical setups, they are all renewed from September to December checking them.

Thanks all!

 

[Mark@EvoIT]

So apparently tons of inconsistency with the start-up script. Not sure who, what, when, or why but apparently they've changed everything from */3cxpbxexpress/ to */3cxpbx/ which is causing mayhem on the start-up scripts on reboot (4x Gcloud 3CX installs, 2 with the first, 1 with the second, 1 mixed throughout). Fixed those under the public key section of the script fired up a clean reboot from ssh and boom past that error, then repo can't find errors below:

Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: W: Failed to fetch http://downloads.3cx.com/downloads/3cxpbxexpress/InRelease
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: W: Failed to fetch http://downloads.3cx.com/downloads/3cxpbxexpress/Release.gpg Unable to connect to www.3cx.com:http:
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: W: Some index files failed to download. They have been ignored, or old ones used instead.
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> Error Code: Err http://downloads.3cx.com InRelease
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> The command failed, retry to update repos.
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> Error Code: Err http://downloads.3cx.com Release.gpg
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> The command failed, retry to update repos.
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> Error Code: 0
Oct 26 21:22:46 pbxexpress startup-script: INFO startup-script: >>>>> The Error Code Detected, retry to update repos.

I found that the repo add on the 3cxpbx.list was causing the error above, making the change to via deb below from /3cxpbxexpress/ to /3cxpbx/ took care of that issue.

echo "deb http://downloads.3cx.com/downloads/3cxpbx/ /" | tee /etc/apt/sources.list.d/3cxpbx.list

Instantly fixed the issue and got console to stop spamming (and causing CPU spikes to 100% every 2-3 seconds up and down).

Now the issue is still SSL certificate expired, any command to force a renewal from terminal? Don't see it in the interface.

Thanks all!

EDIT: Notated clean reboot came up error free no issues, console is squeaky clean and cpu is sitting at next to nothing idle. So now to figure out the SSL issue (console works fine / calls pick up), is there a command to force SSL renewal via terminal? (don't see it in the interface). Does it attempt to renew itself at random intervals each day (check if valid or not, renew is invalid / expired?)

 

[Sopock]

Repo Change
I would check CloudServiceWatcher.log

 

[Mark@EvoIT]

Repo Change
I would check CloudServiceWatcher.log

Thanks for the heads up Sopock! Strange that this is the third deployed PBX, first two have correct repo / config / script entries and are renewing fine (renewed 9/26/17). All three of these were deployed before that point of repo change, strange though the fourth deployed after that date has traces in the scripts of the old repo / link info for keys (seems a little inconsistent from the pbx deploy tool).

I'll give that log a check too see what it says! Thanks for the info for sure

 

[Mark@EvoIT]

------------|Inf(00)|  Date: 10/26/17 9:57:28 PM
2017/10/26 21:57:28.301|360|0001|Inf|Log is created
2017/10/26 21:57:28.591|0001|Info(03)| Service Watcher starting
2017/10/26 21:57:29.120|0001|Info(03)| Next cert renew at: 10/27/17 2:38:37 AM
2017/10/26 23:44:04.744|0005|Info(03)| Instance Instance1 has stopped service(s)
2017/10/26 23:44:04.764|0005|Info(03)| NotifyCloudAdmin: service 3CXQueueManager01 (10.142.0.2) is stopped
2017/10/26 23:44:09.783|0005|Info(03)| Instance Instance1 has stopped service(s)
2017/10/26 23:44:09.783|0005|Info(03)| NotifyCloudAdmin: service 3CXIVR01 (10.142.0.2) is stopped
2017/10/26 23:44:09.783|0005|Info(03)| NotifyCloudAdmin: service 3CXQueueManager01 (10.142.0.2) is stopped
2017/10/26 23:44:14.816|0007|Info(03)| Instance Instance1 has stopped service(s)
2017/10/26 23:44:14.816|0007|Info(03)| NotifyCloudAdmin: service 3CXPhoneSystem01 (10.142.0.2) is stopped
2017/10/26 23:44:14.816|0007|Info(03)| NotifyCloudAdmin: service 3CXIVR01 (10.142.0.2) is stopped
2017/10/26 23:44:14.816|0007|Info(03)| NotifyCloudAdmin: service 3CXQueueManager01 (10.142.0.2) is stopped
2017/10/26 23:44:16.027|0005|Info(03)| Instance Instance1 has stopped service(s)
2017/10/26 23:44:16.027|0005|Info(03)| NotifyCloudAdmin: service 3CXPhoneSystem01 (10.142.0.2) is stopped
2017/10/26 23:44:16.028|0005|Info(03)| NotifyCloudAdmin: service 3CXIVR01 (10.142.0.2) is stopped
2017/10/26 23:44:16.028|0005|Info(03)| NotifyCloudAdmin: service 3CXQueueManager01 (10.142.0.2) is stopped
2017/10/26 23:44:16.028|0005|Info(03)| NotifyCloudAdmin: service 3CXTunnel01 (10.142.0.2) is stopped
2017/10/26 23:45:30.531|0007|Info(03)| Sending email to cloud admin
2017/10/26 23:47:09.647|0001|Info(03)| Service Watcher finishing
------------|Inf(00)|  Date: 10/26/17 11:47:36 PM
2017/10/26 23:47:36.150|359|0001|Inf|Log is created
2017/10/26 23:47:36.406|0001|Info(03)| Service Watcher starting
2017/10/26 23:47:36.571|0001|Info(03)| Next cert renew at: 10/27/17 1:45:12 AM

Log's been pretty spartan aside from failed GDrive backup notices (too many clients, back a bunch of up to a Gsuite account). Looks like since I fixed those scripts before leaving work earlier seemed to do the trick maybe (I'll wait until the renew attempt here in 30 minutes). Thanks for the pointer on where that's logged. I'm curious if you can manually kick it off or if a service does it periodicly.

Strange is that the others that had the script repos wrong, and the repo list outdated are updating and renewing fine go figure lol

 

[Mark@EvoIT]

------------|Inf(00)|  Date: 10/26/17 11:47:36 PM
2017/10/26 23:47:36.150|359|0001|Inf|Log is created
2017/10/26 23:47:36.406|0001|Info(03)| Service Watcher starting
2017/10/26 23:47:36.571|0001|Info(03)| Next cert renew at: 10/27/17 1:45:12 AM
------------|Inf(00)|  Date: 10/27/17 1:45:12 AM
2017/10/27 01:45:12.699|0007|Info(03)| Starting pbxconfigtool to renew certs
2017/10/27 01:46:48.115|0007|Info(03)| Cleaning old global logs task has been started...
2017/10/27 01:46:48.148|0007|Info(03)| Cleaning old global logs task has been finished
2017/10/27 01:46:48.178|0007|Info(03)| Task is finished
2017/10/27 01:46:48.178|0007|Info(03)| Next cert renew at: 10/28/17 2:15:06 AM

Sopock thanks for the pointers on the repo confirmation, think i had it fixed earlier but that let me know to check all my PBX's and update.

Looks like it renewed just fine so all is well, glad it was something simple, strange it's been broken this long (and my older boxes were renewing fine); Strange strange.

 

[YiannisH_3CX]

Glad to see the issue is resolved and thank you for posting your solution.