Toggle sidebar Toggle sidebar

Updating a Debian 9.13 container with 3CX and it hangs trying to restart services

Status
Not open for further replies.

James Crook

Customer
Joined
Jul 24, 2020
Messages
9
Reaction score
0
So after abit of a rabbit hole with kernel messages and proxmox, i'm at the point where if i roll back to the container to 1 Jan it works, then when i do the updates the services fail to start

Code: 
Created symlink /etc/systemd/system/multi-user.target.wants/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.

Created symlink /etc/systemd/system/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.

Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.

Job for 3CXPhoneSystemMC01.service failed because a timeout was exceeded.

See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.

System.Collections.Generic.List`1[System.String]: Some services were not started. You can start them manually.

Created symlink /etc/systemd/system/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.

Created symlink /etc/systemd/system/multi-user.target.wants/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.

Job for 3CXEventNotificationManager.service failed because a fatal signal was delivered to the control process.

See "systemctl status 3CXEventNotificationManager.service" and "journalctl -xe" for details.

System.Collections.Generic.List`1[System.String]: Some services were not started. You can start them manually.

Successfully updated

Setting up postgresql-contrib-9.6 (9.6.17-0+deb9u1) ...

Processing triggers for libc-bin (2.24-11+deb9u4) ...

Processing triggers for ca-certificates (20200601~deb9u1) ...

Updating certificates in /etc/ssl/certs...

0 added, 0 removed; done.

Running hooks in /etc/ca-certificates/update.d...

done.

These are the updates it listed

Code: 
The following packages will be upgraded:

  3cxpbx apt apt-utils base-files bind9-host ca-certificates dbus dnsutils libapt-inst2.0 libapt-pkg5.0

  libbind9-140 libcurl3 libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libexif12 libfreetype6

  libglib2.0-0 libglib2.0-data libgnutls-openssl27 libgnutls30 libicu57 libidn11 libirs141 libisc-export160

  libisc160 libisccc140 libisccfg140 libldap-2.4-2 libldap-common liblwres141 libnginx-mod-http-auth-pam

  libnginx-mod-http-dav-ext libnginx-mod-http-echo libnginx-mod-http-geoip libnginx-mod-http-image-filter

  libnginx-mod-http-subs-filter libnginx-mod-http-upstream-fair libnginx-mod-http-xslt-filter

  libnginx-mod-mail libnginx-mod-stream libperl5.24 libpq5 libpython3.5-minimal libpython3.5-stdlib

  libsasl2-2 libsasl2-modules-db libssl1.0.2 libtiff5 libxslt1.1 nginx nginx-common nginx-full perl perl-base

  perl-modules-5.24 postfix postfix-sqlite postgresql postgresql-9.6 postgresql-client-9.6

  postgresql-client-common postgresql-common postgresql-contrib-9.6 python-apt-common python-cryptography

  python3-apt python3.5 python3.5-minimal sudo tzdata

This as after complaining about salt stack

Code: 
Err:7 http://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch Release

  404  Not Found [IP: 99.86.116.69 80]

Reading package lists... Done                     

E: The repository 'http://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch Release' does no longer have a Release file.

N: Updating from such a repository can't be done securely, and is therefore disabled by default.

N: See apt-secure(8) manpage for repository creation and user configuration details.
 
Hi,

Please run the following command and then try to update again from the management console

Bash: 
wget -O - https://downloads.3cx.com/downloads/misc/debianpatches/9/16/0/5/20200619-os-update.sh | bash
 
  • Like
Reactions: AlexooSlovakia
JohnS_3CX said:
Hi,

Please run the following command and then try to update again from the management console

Bash: 
wget -O - https://downloads.3cx.com/downloads/misc/debianpatches/9/16/0/5/20200619-os-update.sh | bash
Click to expand...

That fixed the salt complaining issues, but still hangs at 98%

Code: 
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystem01.service → /lib/systemd/system/3CXPhoneSystem01.service.
Created symlink /etc/systemd/system/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXCallFlow01.service → /lib/systemd/system/3CXCallFlow01.service.
Created symlink /etc/systemd/system/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXIVR01.service → /lib/systemd/system/3CXIVR01.service.
Created symlink /etc/systemd/system/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXQueueManager01.service → /lib/systemd/system/3CXQueueManager01.service.
Created symlink /etc/systemd/system/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXSystemService01.service → /lib/systemd/system/3CXSystemService01.service.
Created symlink /etc/systemd/system/3CXTunnel01.service → /lib/systemd/system/3CXTunnel01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXTunnel01.service → /lib/systemd/system/3CXTunnel01.service.
Created symlink /etc/systemd/system/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXAudioProvider01.service → /lib/systemd/system/3CXAudioProvider01.service.
Created symlink /etc/systemd/system/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXPhoneSystemMC01.service → /lib/systemd/system/3CXPhoneSystemMC01.service.

Progress: [ 98%] [#######################################################################################..

Once it reaches the timeout of the service it says

Code: 
System.Collections.Generic.List`1[System.String]: Some services were not started. You can start them manually.
Created symlink /etc/systemd/system/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Created symlink /etc/systemd/system/multi-user.target.wants/3CXEventNotificationManager.service → /lib/systemd/system/3CXEventNotificationManager.service.
Job for 3CXEventNotificationManager.service failed because a fatal signal was delivered to the control process.
See "systemctl status 3CXEventNotificationManager.service" and "journalctl -xe" for details.
System.Collections.Generic.List`1[System.String]: Some services were not started. You can start them manually.
Successfully updated
Setting up postgresql-contrib-9.6 (9.6.17-0+deb9u1) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for ca-certificates (20200601~deb9u1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
 
Try to restart and intercept the next error
 
JohnS_3CX said:
Try to restart and intercept the next error
Click to expand...

back to timeout on the service

Code: 
Jul 24 14:45:56 intralan3cx systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Jul 24 14:47:56 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Start operation timed out. Terminating.
Jul 24 14:47:56 intralan3cx systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
Jul 24 14:47:56 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Jul 24 14:47:56 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'timeout'

even tried another apt update && apt dist-upgrade
but there were no packages to update.
 
Upgrades are not the end all be all, it's best to see why the service times out (and usually there is a good reason).

If you notice higher up after the service timed out there was a message saying:
1595599408894.png
 
q = quit in journalctl
 
Code: 
Job for 3CXPhoneSystemMC01.service failed because a timeout was exceeded.
See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
root@intralan3cx:~#
root@intralan3cx:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Fri 2020-07-24 15:15:25 BST; 36s ago
  Process: 827 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid
 Main PID: 827 (code=killed, signal=INT)

Jul 24 15:13:25 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed to reset devices.list: Operation not
Jul 24 15:13:25 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed to set invocation ID on control grou
Jul 24 15:13:25 intralan3cx systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
Jul 24 15:15:25 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Start operation timed out. Terminating.
Jul 24 15:15:25 intralan3cx systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
Jul 24 15:15:25 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Jul 24 15:15:25 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'timeout'.

I can post the output of journalctl -xe, but its very long.

also i can see all the other 3CX servers are starting fine
 
run a netstat -laputen | grep "5001" provided that your management console was originally configured to run on 5001

Bash: 
root@js-debian:~# netstat -laputen | grep "5001"
tcp        0      0 0.0.0.0:5001            0.0.0.0:*               LISTEN      0          12669      679/nginx: master p
tcp        0      0 10.31.0.46:5001         10.31.0.146:60180       ESTABLISHED 33         597048     680/nginx: worker p
tcp        0      0 10.31.0.46:5001         10.31.0.146:60181       ESTABLISHED 33         597106     680/nginx: worker p
tcp6       0      0 :::5001                 :::*                    LISTEN      0          12670      679/nginx: master p
root@js-debian:~#

If something else is running on that port then the MC cannot start and it will time out.
 
JohnS_3CX said:
run a netstat -laputen | grep "5001" provided that your management console was originally configured to run on 5001

Bash: 
root@js-debian:~# netstat -laputen | grep "5001"
tcp        0      0 0.0.0.0:5001            0.0.0.0:*               LISTEN      0          12669      679/nginx: master p
tcp        0      0 10.31.0.46:5001         10.31.0.146:60180       ESTABLISHED 33         597048     680/nginx: worker p
tcp        0      0 10.31.0.46:5001         10.31.0.146:60181       ESTABLISHED 33         597106     680/nginx: worker p
tcp6       0      0 :::5001                 :::*                    LISTEN      0          12670      679/nginx: master p
root@js-debian:~#

If something else is running on that port then the MC cannot start and it will time out.
Click to expand...
root@intralan3cx:~# netstat -laputen | grep "5001"
tcp 0 0 0.0.0.0:5001 0.0.0.0:* LISTEN 0 1147965409 250/nginx: master p
tcp6 0 0 :::5001 :::* LISTEN 0 1147965410 250/nginx: master p

I can web to it and get the login, page, just says server error when i press the login button
 
restored it again, was working fine, then it stopped working again (i ran your script and after a few seconds it stopped working)

i caught this in the logs

Code: 
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Fri 2020-07-24 15:57:40 BST; 5min ago
  Process: 1904 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 1903 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Main PID: 425 (code=killed, signal=SEGV)

Jul 24 15:51:20 intralan3cx sudo[658]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 24 15:51:20 intralan3cx sudo[689]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/sbin/install-instance-manager 1 1 1
Jul 24 15:51:20 intralan3cx sudo[689]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 24 15:51:21 intralan3cx sudo[746]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/bin/apt-get update -o Dir::Etc::sourcelist=sources.list.d/3cxpbx-testing.list -o Dir::Etc::sourceparts=- -o APT::Get::List-Cleanup=0
Jul 24 15:51:21 intralan3cx sudo[746]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 24 15:51:21 intralan3cx systemd[1]: Started 3CX PhoneSystem 01 Management Console.
Jul 24 15:57:40 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=11/SEGV
Jul 24 15:57:40 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed to reset devices.list: Operation not permitted
Jul 24 15:57:40 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
Jul 24 15:57:40 intralan3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.

Looks like its auto updating and after that it won't start the service
 
Is there anyway to stop this auto update feature ?, short of blocking all out bound traffic (which i think then causes 3CX not to start as it can't check in)
 
Hi James,

The "apt update" command does not literally update any components, it just updates the list of programs so it knows what versions exist in the repository vs the installed versions.

I'm not sure if the container environment is affecting the system in any way, officially we support installation directly on a Debian OS only (although containers are theoretically possible if properly tested).
If you don't have very good knowledge of container environments (and I mean on an advanced level) then you may be better off installing it directly on a Debian OS on a dedicated machine or VM.

Use a supported method of running the PBX that is known to be good, so you can have stable installation and avoid containers for now.
The 3CX ISO is perfect for this purpose, just read carefully to ensure you are running in a supported environment: https://www.3cx.com/docs/manual/installing-debian-linux-pbx/
 
JohnS_3CX said:
Hi James,

The "apt update" command does not literally update any components, it just updates the list of programs so it knows what versions exist in the repository vs the installed versions.

I'm not sure if the container environment is affecting the system in any way, officially we support installation directly on a Debian OS only (although containers are theoretically possible if properly tested).
If you don't have very good knowledge of container environments (and I mean on an advanced level) then you may be better off installing it directly on a Debian OS on a dedicated machine or VM.

Use a supported method of running the PBX that is known to be good, so you can have stable installation and avoid containers for now.
The 3CX ISO is perfect for this purpose, just read carefully to ensure you are running in a supported environment: https://www.3cx.com/docs/manual/installing-debian-linux-pbx/
Click to expand...

true, upgrade would upgrade the packages. But it's rather strange that after it has called that it then restarts the service. at which point it stops working, when it was working before.

Many be it had queued an update and applies it on first start, then checks for the lastest version ?
"COMMAND=/usr/sbin/install-instance-manager"
 
The instance manager is not required for the main system to come up.

It's a separate service that can run on the side or even be disabled entirely.

I think you should try like I mentioned above, and use a supported environment rather than spend a considerable amount of time hunting down the root cause, only to find another issue down the road.
 
  • Like
Reactions: James Crook
I agree with you, we are investigating Azure and other full virtual environments.
 
Azure should be fine as well as AWS, LightSail, OHV, Google.

You can optionally deploy using PBX Express on those platforms (automated VM creation and installation) for an extremely fast deployment using an existing backup even.
https://www.3cx.com/docs/cloud-pbx-providers/
 
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,955
Messages
589,926
Members
164,855
Latest member
parik24pro
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom