Services Not Starting PostgreSQL, 3CXPhoneStstemMC01.service

[jrb]

Hi All,

I have a 3CX Debian install that isn't starting up properly.

Getting a few errors about some services not starting due to a 'Protocol' error.
-- The unit 3CXCfgServ01.service has entered the 'failed' state with result 'protocol'.
Jan 10 07:05:25 ag3cx01 systemd[1]: Failed to start 3CX PhoneSystem Configuration Service.
-- Subject: A start job for unit 3CXCfgServ01.service has failed
-- Defined-By: systemd

The unit [email protected] has entered the 'failed' state with result 'protocol'.
Jan 10 07:02:55 ag3cx01 systemd[1]: Failed to start PostgreSQL Cluster 11-main.
-- Subject: A start job for unit [email protected] has failed
-- Defined-By: systemd

full logs attached.

Server has been restarted a couple times.
I have it a little extra disk space and RAM in case that was being an issue but that hasn't helped.

 

[jrb]

As far as I can tell the issue might be with the PostgreSQL config file:
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_messages": "en_AU.UTF-8"
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_monetary": "en_AU.UTF-8"
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_numeric": "en_AU.UTF-8"
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_time": "en_AU.UTF-8"
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] FATAL: configuration file "/etc/postgresql/11/main/postgresql.conf" contains errors
Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: pg_ctl: could not start server

 

[jrb]

I've fixed the issue for now. For any future people having the same issue or who are interested here is my condenced troubleshooting steps:

This is a Debian Server privately hosted.

Initially the whole server was not responding so I performed a hard reset on it.

Once I could get console access I did the following:

systemctl start 3CXPhoneServerMC01.service

Which failed because of a ‘dependency’ error. It told me to run another command to view the logs:

Journalctl -xe

Doing this showed the [email protected] was failing to start.

I concentrated on this service as I thought it must mean the system database wasn’t starting.



After much troubleshooting/Googling and research, I found the following:

Systemctl status [email protected]

That command showed me there was an issue in the language/locale pack:

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_messages": "en_AU.UTF-8"

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_monetary": "en_AU.UTF-8"

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_numeric": "en_AU.UTF-8"

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] LOG: invalid value for parameter "lc_time": "en_AU.UTF-8"

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: 2023-01-10 08:29:09.319 +08 [2130] FATAL: configuration file "/etc/postgresql/11/main/postgresql.conf" contains errors

Jan 10 08:29:09 ag3cx01 postgresql@11-main[2125]: pg_ctl: could not start server



More googling found I could install the Locale pack:

Sudo locale-gen en_AU.utf8

Reran the status check again and found a new error:

Jan 10 08:47:06 ag3cx01 systemd[1]: Starting PostgreSQL Cluster 11-main...

Jan 10 08:47:06 ag3cx01 postgresql@11-main[548]: Error: /usr/lib/postgresql/11/bin/pg_ctl /usr/lib/postgresql/11/bin/pg_ctl start -D /var/lib/postgresql/11/main -l /var/log/postgresql/postg

resql-11-main.log -s -o -c config_file="/etc/postgresql/11/main/postgresql.conf" exited with status 1:

Jan 10 08:47:06 ag3cx01 postgresql@11-main[548]: 2023-01-10 08:47:06.268 +08 [553] FATAL: data directory "/var/lib/postgresql/11/main" has invalid permissions

Jan 10 08:47:06 ag3cx01 postgresql@11-main[548]: 2023-01-10 08:47:06.268 +08 [553] DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

Jan 10 08:47:06 ag3cx01 postgresql@11-main[548]: pg_ctl: could not start server

Jan 10 08:47:06 ag3cx01 postgresql@11-main[548]: Examine the log output.

Jan 10 08:47:06 ag3cx01 systemd[1]: [email protected]: Can't open PID file /run/postgresql/11-main.pid (yet?) after start: No such file or directory

Jan 10 08:47:06 ag3cx01 systemd[1]: [email protected]: Failed with result 'protocol'.

Jan 10 08:47:06 ag3cx01 systemd[1]: Failed to start PostgreSQL Cluster 11-main.

So the issue is a permissions issue to /var/lig/postgresql/11/main. Fixed this using:
chmod 700 /var/lib/postgresql/11/main

Now I could start the sql service:
systemctl start [email protected]

It started but the management console was down.
did a restart on the server, still down.

I manually started the Management Console:
systemctl start 3CXPhoneServerMC01.service

This got the web console up, where I could then start the rest of the stopped services.

 

[Nathan@Voxtelesys]

Troubleshooting an issue with postgres is kind of a rabbit trail. Your quickest way to be up and running again is by restoring a backup to a fresh install. A bonus to this is you will be up-to-date if you were not previously.
Do you have a backup?

If you want to try something adventurous and have your 3CX data backed up in a .zip, you could try re-installing 3CX in place with apt-get --reinstall install 3cxpbx

Had you initially ran out of storage space? As for the RAM increase, match the spec recommendations here: https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/

 

[Nathan@Voxtelesys]

I've fixed the issue for now. For any future people having the same issue or who are interested here is my condenced troubleshooting steps:

This got the web console up, where I could then start the rest of the stopped services.

Yea.... You probably want to take a backup from the MGMT Console and re-install on the official ISO download in the 3CX Portal. You likely still have an issue on your hands. Even if it is running for now -- it may not reboot on its own.

Manually starting postgres/3CX services and installing locales is not typical behavior at all. By using the official ISO, these odd locale issues probably won't happen.

 

[jrb]

Yes we have a good backup, so we will plan to rebuild the server.
It was pretty much the case of, "oh, i think this is the issue" then kept going down the rabbit hole.
I agree that a restore from backup would have been quicker.

Thanks for your reply!