'Server Error' when logging into PBX - all services showing up in SSH

[pop123]

Hello,

Got a strange issue when trying to log into server web interface, it throws 'Server Error', however the PBX is online, all extensions are connected, calls coming through and all services show online when queried via SSH. All updates performed via SSH too with no changes.

Any advice appreciated.

Currently running V18 as can't login to update to V20!

Thanks in advance.

 

[Guillaume Bourgeois]

If possible and if 3CX is running on a VM, take a snapshot. Once the snapshot is completed, connect to the server via SSH and run the following command:

systemctl list-units --failed

It should display any failed services. Take a screenshot of all the failed services.

If `3CXPhoneSystemMC01` is listed as failed (which it likely will be), run the following command:

systemctl restart 3CXPhoneSystemMC01

If everything returns to normal, proceed with a system reboot for 3CX to ensure that everything remains stable during the upgrade to v20.

If any issues arise, revert to your snapshot, and we can investigate further to identify the cause.

 

[pop123]

Hi, Thanks for the reply.

Command ran:

0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

* when running below, it shows up.....
- systemctl status 3CXPhoneSystemMC01
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vend
or preset: enabled)
Active: active (running) since Fri 2024-08-02 19:16:11 BST; 2h 29m
in ago
Main PID: 40836 (3CXManagementCo)
Tasks: 45 (limit: 4660)
Memory: 309.4M
CGroup: /system.slice/3CXPhoneSystemMC01.service
└─40836 /usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesyst
em/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot

As mentioned, all services are showing as up - it's bizarre. I can't work out why it's not allowing me in via the web interface. It was fine one day and then it wasn't. I'd prefer not to have to take the client offline to revert to a backup.

Any further thoughts?

Many thanks

 

[Guillaume Bourgeois]

Try restar service ;

systemctl restart 3CXPhoneSystemMC01

It's most likely just a small bug.

 

[pop123]

Sorry for delayed response. Executed - brings back same result unfortunately.

 

[Guillaume Bourgeois]

Try ;

sudo systemctl restart nginx

If that doesn't work either:

Check if there's enough space on your hard drive:

df -h

If there's enough space (at least 50%):

Don't overthink it.
Proceed to back up using the following SSH commands:

sudo -u phonesystem 3CXBackupCmd --file=/var/tmp/full_pbx_backup.zip --options=ALL --log=/var/tmp/pbx-backup_cmd.log

Export the backup, and start fresh with a new 3CX v20 installation without any configuration.
Restore your v18 backup.

Good luck

 

[pop123]

nginx restart didn't work unfortunately.

Plenty of free space

Error on the backup - could be command related though:

find: Failed to restore initial working directory: /root: Permission denied

 

[Guillaume Bourgeois]

nginx restart didn't work unfortunately.

Plenty of free space

Error on the backup - could be command related though:

find: Failed to restore initial working directory: /root: Permission denied

sudo -u phonesystem 3CXBackupCmd --file=/var/lib/3cxpbx/Instance1/Data/Backups/full_pbx_backup.zip --options=ALL --log=/var/lib/3cxpbx/Instance1/Data/Backups/pbx-backup_cmd.log

Try using this command (with the official backup directory).

Do not change the directory.

 

[pop123]

Still get the same error unfortunately.

 

[Guillaume Bourgeois]

Still get the same error unfortunately.

Send me in private message the pbx-backup_cmd.log

 

[Guillaume Bourgeois]

If you don't have a log file at /var/lib/3cxpbx/Instance1/Data/Backups/pbx-backup_cmd.log,

Run the following command and try again.

sudo chown -R phonesystem:phonesystem /var/lib/3cxpbx

 

[pop123]

Okay, managed to sort the backup. Thank you.

Last question before doing the reinstallation of V20 - if I don't have a user with 'Owner Rights' (and i obviously can't set one as i can't login to the interface!), will that cause an issue?

Many thanks for your help with this.

 

[Evolute IT]

Okay, managed to sort the backup. Thank you.

Last question before doing the reinstallation of V20 - if I don't have a user with 'Owner Rights' (and i obviously can't set one as i can't login to the interface!), will that cause an issue?

Many thanks for your help with this.

Yes this is a problem. It was indicated everywhere in blog posts and the update dialog itself that you need a system owner. If you haven't created one, you'll need to go into the backup XML (if you have one) and change some values manually. There's a post from a few days ago explaining how to do this on a thread somewhere. Can't find it at the moment.

 

[Guillaume Bourgeois]

Okay, managed to sort the backup. Thank you.

How did you resolve the backup issue?

Perhaps you addressed the web interface problem at the same time? Try restarting the services again and see the result.

 

[pop123]

How did you resolve the backup issue?

Perhaps you addressed the web interface problem at the same time? Try restarting the services again and see the result.

I typed the command in correctly (on the second line you sent)

Web interface is still a problem - so I've got no choice but to wipe it I suspect. Just need to work out how to deal with the Owner Rights issue now.

Many thanks.

 

[Guillaume Bourgeois]

Modifying a user in 3CX without the web interface will be nearly impossible.
That said...

Here are some other avenues to explore:

After the command that repaired the backups,
did you restart the entire VM (not just the 3CX services)?

Once done, try using another computer (or use private browsing mode) to attempt to connect to the web interface.

Still not working? Same error?

Send me the FQDN of your phone system , in a private message.
I don't need the username and password, just the FQDN.

I will inspect the Google Chrome console while I attempt to connect and see if there are any clues in the console explaining the error.

 

[Guillaume Bourgeois]

After a private discussion with @pop123 , I'm sharing our collaboration publicly as it may be beneficial for others and resolved their problem.

@pop123 discovered a discussion detailing similar issues on Windows.
See https://www.3cx.com/community/threads/server-error-trying-to-log-in-error-reading-key-ring.60168/
He shared the details of the solution with me and asked if there was an equivalent available in Linux.

I warned him that manipulating the 3CX files could worsen the problem. I had never performed this operation before. @pop123 agreed to execute the procedure (adapted for Linux) despite the potential risks involved. .

I sent him the command to copy files from /var/lib/3cxpbx/.aspnet/ and to delete the files;
In the end, it looks like this:

sudo mkdir -p /root/3cxpbx/.aspnet/ && sudo rsync -avzh /var/lib/3cxpbx/.aspnet/ /root/3cxpbx/.aspnet/ && sudo find /var/lib/3cxpbx/.aspnet/ -type f -exec rm {} + && echo "Please wait ..." && sudo systemctl restart 3CXPhoneSystemMC01 && echo "Completed! Try logging into the 3CX Phone System web page." || echo "An error occurred."

Here's a description of what each command in the sequence does:

1. sudo mkdir -p /root/3cxpbx/.aspnet/:
- Creates the directory /root/3cxpbx/.aspnet/, including any necessary parent directories. The sudo ensures it runs with administrative privileges.

2. sudo rsync -avzh /var/lib/3cxpbx/.aspnet/ /root/3cxpbx/.aspnet/:
- Synchronizes files from /var/lib/3cxpbx/.aspnet/ to /root/3cxpbx/.aspnet/, preserving attributes like permissions and timestamps. sudo is used for permissions, -a ensures archival mode, -v for verbose output, -z compresses file data during transfer, and -h displays sizes in a human-readable format.

3. sudo find /var/lib/3cxpbx/.aspnet/ -type f -exec rm {} +:
- Searches for all files (type f) in /var/lib/3cxpbx/.aspnet/ and deletes each one (exec rm {} +). The sudo ensures it runs with root permissions.

4. echo "Please wait ...":
- Outputs "Please wait ..." to the terminal, providing a user-friendly message indicating that further actions are being processed.

5. sudo systemctl restart 3CXPhoneSystemMC01:
- Restarts the service 3CXPhoneSystemMC01, using sudo to ensure it has the necessary permissions.

6. echo "Completed! Try logging into the 3CX Phone System web page.":
- Outputs this message upon successful completion of all previous commands, suggesting that the user tries logging into the 3CX system web page.

7. || echo "An error occurred.":
- If any command preceding the || fails, it outputs "An error occurred." This is used for basic error handling to inform the user if something went wrong during the execution.

Glad to have helped solve the problem! Thanks for your patience @pop123 .​

 

[pop123]

Glad to have helped solve the problem! Thanks for your patience @pop123 .

Without you and the continuous troubleshooting, I'd have gotten nowhere, so again, you're the savior! Thank you again!

Hopefully this helps someone else who comes across this rather bizarre issue with a fully functioning PBX you can't login to via the web interface!