Half of v18-v20 systems restored from backup to new GCP 3CX template v20 are crashing after 24hrs.

[copetek]

Half of v18-v20 systems restored from backup to new GCP 3CX template v20 are crashing after 24hrs.

I rebuilt about 15 3cx systems over the weekend.

I created a new 3CX system from the GCP 3cx v20 template. Took a backup of the old v18 system, shut both down and swapped the public IP. Then booted the new system up and restored the backup. This works great and have done this many times on previous versions. Ran all my checks and the systems were up and stable, plenty of free space on disk storage and system / swap memory. The next day 3 of the systems were no longer running. I SSH into them and Debian 12 is running fine but the 3cx service wont start. Rebooted and same result, the 3cx service wont start. See Picture.

So after researching and not finding anything to help. I just reloaded the machines again and it was back up and running. Then the next day 3 more systems, different ones did the same thing.

Error from syslog
2024-06-24T10:33:40.301324-05:00 localhost systemd[1]: Starting 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service...
2024-06-24T10:34:05.321014-05:00 localhost systemd[1]: 3CXCfgServ01.service: Failed with result 'protocol'.
2024-06-24T10:34:05.321612-05:00 localhost systemd[1]: Failed to start 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service.
2024-06-24T10:34:05.321744-05:00 localhost systemd[1]: Dependency failed for 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console.
2024-06-24T10:34:05.322085-05:00 localhost systemd[1]: 3CXPhoneSystemMC01.service: Job 3CXPhoneSystemMC01.service/start failed with result 'dependency'.



I then opened a ticket with 3cx and they told me they don't support operating systems. See message below. This is not an operating system error, this is something within the 3cx service that is failing. I m trying to locate additional logs to help troubleshooot the service but need some help. I have since reloaded those machines again to get them up, but I fear more will crash.



"Hello Chad,

As per 3CX Support Policies and Procedures that you can see here, we do not offer support for Operating System-related issues.
If you continue to experience issues with your systems we advise that you provision them using the 3CX Hosting and restore a backup to it, if the problem persists we will then be able to assist you further in investigating and resolving it. Note that telephone support is only available for partners that are Silver Partners or higher.
Additionally, ensure to follow the 3CX guide when provisioning the 3CX PBX from Google Cloud Marketplace.

Note: You can reply to the case by sending emails from "[email protected]" and keeping the same email subject, or from your Customer Portal / Support page.
---
Best regards,
Renato Alvares
Level 1 Engineer

For Opening Hours (Monday to Friday) log in to https://portal.3cx.com"

 

[Alphabetic]

if you do

df -h

do you have enough disk space?

 

[copetek]

Yes, I have ran into that issue in the past so that was my first thought.

Filesystem Size Used Avail Use% Mounted on
udev 979M 0 979M 0% /dev
tmpfs 199M 456K 198M 1% /run
/dev/sda1 30G 4.8G 24G 18% /
tmpfs 991M 1.1M 990M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 124M 12M 113M 10% /boot/efi
tmpfs 199M 0 199M 0% /run/user/1001

 

[Alphabetic]

that boot/efi suggests youre running UEFI bios, I dont believe the set up is compatible - could be a red herring, could be the cause.

are you able to set up VM with legacy?

 

[copetek]

I deployed the google cloud template from 3cx for v20. This was the official configuration that 3cx setup.

 

[Alphabetic]

what happens if you type

journalctl -xe

 

[copetek]

Thats whats posted in the image attached. Its saying a dependency failed, but having trouble finding a log that can give more information.

Jun 24 10:46:34 integritym-3cx-bookworm-june2024-vm systemd[1]: Starting 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service...
░░ Subject: A start job for unit 3CXCfgServ01.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit 3CXCfgServ01.service has begun execution.
░░
░░ The job identifier is 2123.
Jun 24 10:46:46 integritym-3cx-bookworm-june2024-vm sshd[2609]: Received disconnect from 201.71.21.1 port 50888:11: Bye Bye [preauth]
Jun 24 10:46:46 integritym-3cx-bookworm-june2024-vm sshd[2609]: Disconnected from authenticating user root 201.71.21.1 port 50888 [preauth]
Jun 24 10:46:57 integritym-3cx-bookworm-june2024-vm sshd[2612]: Received disconnect from 187.200.216.4 port 57572:11: Bye Bye [preauth]
Jun 24 10:46:57 integritym-3cx-bookworm-june2024-vm sshd[2612]: Disconnected from authenticating user root 187.200.216.4 port 57572 [preauth]
Jun 24 10:46:59 integritym-3cx-bookworm-june2024-vm systemd[1]: 3CXCfgServ01.service: Failed with result 'protocol'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit 3CXCfgServ01.service has entered the 'failed' state with result 'protocol'.
Jun 24 10:46:59 integritym-3cx-bookworm-june2024-vm systemd[1]: Failed to start 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service.
░░ Subject: A start job for unit 3CXCfgServ01.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit 3CXCfgServ01.service has finished with a failure.
░░
░░ The job identifier is 2123 and the job result is failed.
Jun 24 10:46:59 integritym-3cx-bookworm-june2024-vm systemd[1]: Dependency failed for 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console.
░░ Subject: A start job for unit 3CXPhoneSystemMC01.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit 3CXPhoneSystemMC01.service has finished with a failure.
░░
░░ The job identifier is 2045 and the job result is dependency.
Jun 24 10:46:59 integritym-3cx-bookworm-june2024-vm systemd[1]: 3CXPhoneSystemMC01.service: Job 3CXPhoneSystemMC01.service/start failed with result 'dependency'.
Jun 24 10:46:59 integritym-3cx-bookworm-june2024-vm sudo[2604]: pam_unix(sudo:session): session closed for user root

 

[Alphabetic]

Failing that, you'll have to pull the OS logs from /var/log and look at syslog

 

[copetek]

This is what that log shows below. I see this error "Error: Could not open file "/var/lib/3cxpbx/Bin/nftables.conf": No such file or directory" any insight?

2024-06-24T10:45:04.667765-05:00 localhost systemd[1]: Starting 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service...
2024-06-24T10:45:29.708713-05:00 localhost systemd[1]: 3CXCfgServ01.service: Failed with result 'protocol'.
2024-06-24T10:45:29.709108-05:00 localhost systemd[1]: Failed to start 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service.
2024-06-24T10:45:29.709854-05:00 localhost systemd[1]: Dependency failed for 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console.
2024-06-24T10:45:29.709994-05:00 localhost systemd[1]: 3CXPhoneSystemMC01.service: Job 3CXPhoneSystemMC01.service/start failed with result 'dependency'.
2024-06-24T10:46:34.187844-05:00 localhost systemd[1]: Starting 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service...
2024-06-24T10:46:59.228635-05:00 localhost systemd[1]: 3CXCfgServ01.service: Failed with result 'protocol'.
2024-06-24T10:46:59.229745-05:00 localhost systemd[1]: Failed to start 3CXCfgServ01.service - 3CX PhoneSystem Configuration Service.
2024-06-24T10:46:59.229862-05:00 localhost systemd[1]: Dependency failed for 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console.
2024-06-24T10:46:59.229942-05:00 localhost systemd[1]: 3CXPhoneSystemMC01.service: Job 3CXPhoneSystemMC01.service/start failed with result 'dependency'.
2024-06-24T10:48:50.793262-05:00 localhost systemd[1]: Starting 3CXFirewall.service - 3CX PhoneSystem Firewall...
2024-06-24T10:48:50.796477-05:00 localhost nft[2676]: internal:0:0-0: Error: Could not open file "/var/lib/3cxpbx/Bin/nftables.conf": No such file or directory
2024-06-24T10:48:50.796806-05:00 localhost systemd[1]: 3CXFirewall.service: Main process exited, code=exited, status=1/FAILURE
2024-06-24T10:48:50.796971-05:00 localhost systemd[1]: 3CXFirewall.service: Failed with result 'exit-code'.
2024-06-24T10:48:50.797280-05:00 localhost systemd[1]: Failed to start 3CXFirewall.service - 3CX PhoneSystem Firewall.

 

[copetek]

So comparing two systems both v20, one that is crashed and the other that is running. In directory "/var/lib/3cxpbx/Bin" the crashed system is missing 3CXGatewayService.config.json, nftables.conf, 3cxmediaserver.ini in the directory.

 

[Alphabetic]

Hmm.. interesting you've got the same set ups and similar issues.

I would maybe see if 3CX can escalate, after all if you're using their image on their supported requirements I see no reason as to why they'd deny support.

 

[copetek]

I tried they just flat out deny support and has ignored my responses to the ticket. Thats when I came to the forum, I am blown away by the support response and quite irritated by it.

 

[Alphabetic]

Try moving one to OVH and follow my guide https://www.3cx.com/community/threads/how-to-install-3cx-iso-on-ovhs-vps.124149/

 

[copetek]

I would prefer to stick to the official template. For the convenience of reloading machines and deploying new systems.

 

[copetek]

So I attached the snapshot from the night before and the directory "/var/lib/3cxpbx/Bin" has the files that were missing in it. I copied them to the production machine and the 3cx service still failed to start. However, when I try to start the 3CXFirewall.service that will start now. So I am thinking some other files within 3cx are missing and not sure how that would happen unless there is some script that runs in 3cx that is causing problems. I didn't take a backup of the system while it was up and running on v20 so I don't think its the nightly backup.

 

[copetek]

So this is concerning, in that I have no support from 3CX, I just tried calling the sales rep but went straight to voicemail. I did send him an email, maybe this can get to someone that can help. I fear I am going to be battling random 3cx v20 systems crashing and the only fix so far is to re-image them and restore the latest backup.

 

[copetek]

Final response from 3CX Official Support. They don't seem to care that there could be an issue with v20, they will only support you if you are on 3cx hosting. So that is their only suggestion, move to 3cx hosting. 3CX used to be great and provide excellent support in the past. But after this, I will have to rethink on weather or not I will continue to use 3cx in my phone system offerings. I am just blown away by the lack of care 3cx provides especially considering the google template was one that is provided by 3cx.

Hello Chad,

It does not appear to be a 3CX-related issue as well if you monitor/inspect the journalctl logs, you may find additional information.

Please let me know if you migrate the server to a hosted by 3CX instance and the issue happens again.

Ensure that the V18 backup file you are using to restore during the installation of a V20 PBX is on the latest available version and be sure to check the V20 Upgrade Checklist & FAQ and the V20 Version Upgrades - Top Reasons for Aborting blog posts to ensure that the systems are configured as required before the upgrade.

Note: You can reply to the case by sending emails from "[email protected]" and keeping the same email subject, or from your Customer Portal / Support page.
---
Best regards,
Renato Alvares
Level 1 Engineer

For Opening Hours (Monday to Friday) log in to https://portal.3cx.com

 

[Alphabetic]

I would prefer to stick to the official template. For the convenience of reloading machines and deploying new systems.

technically, the link I sent you is a supported way as you're using the ISO.

 

[copetek]

Right now all the systems are still running. But if this occurs again, I will just setup a Debian machine and manually install 3cx like we did before the templates. 3CX support said they will look into this further. I was also noticing one specific IP address trying to login to multiple systems. Maybe there is a vulnerability in the software that has been exploited. I since added that IP to my network blacklist.