Toggle sidebar Toggle sidebar

Services crash on V18

Status
Not open for further replies.

bigdessert

Silver Partner
Advanced Certified
Joined
Apr 3, 2017
Messages
90
Reaction score
31
We upgraded from 16 to 18 last night on our PBX going from Debian 9 to 10. At 1:52AM we received the alert

The following service(s) were interrupted:
3CXGatewayService
3CXPhoneSystemMC01

I checked and there is a core file in /var/3cxpbx
I also checked syslog and see this entry at that timeframe

Aug 18 01:51:35 pbx systemd[1]: 3CXGatewayService.service: Main process exited, code=killed, status=6/ABRT
Aug 18 01:51:35 pbx systemd[1]: 3CXGatewayService.service: Failed with result 'signal'.
Aug 18 01:51:36 pbx systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Aug 18 01:51:37 pbx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.

Any suggestions on what could be causing this? I have since restarted the server to get everything back up and running and will monitor if this happens again.
 
Hi @bigdessert,

Did the crash occur during the upgrading from Deb 9 - 10 or after it was complete?
 
The upgrade completed about 10PM without errors and everything appeared to work. This happened in the middle of the night.
 
Hi @bigdessert,

Your syslog should have more data for you to check. How much memory have you allocated to the system? Do you see any reference to 'OOM' in the syslog?
 
Here is the syslog leading up to the issue and after until we rebooted this morning. No OOM errors.

Code: 
Aug 18 00:00:26 pbx rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="407" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Aug 18 00:00:28 pbx rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="407" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Aug 18 00:00:28 pbx systemd[1]: logrotate.service: Succeeded.
Aug 18 00:00:28 pbx systemd[1]: Started Rotate log files.
Aug 18 00:01:07 pbx systemd[1]: man-db.service: Succeeded.
Aug 18 00:01:07 pbx systemd[1]: Started Daily man-db regeneration.
Aug 18 00:05:01 pbx CRON[23404]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 00:15:01 pbx CRON[23757]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 00:17:01 pbx CRON[23828]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 00:25:01 pbx CRON[24139]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 00:35:01 pbx CRON[24493]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 00:45:01 pbx CRON[24830]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 00:55:01 pbx CRON[25175]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:05:01 pbx CRON[25509]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:06:43 pbx kernel: [11935.946835] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module.
Aug 18 01:15:01 pbx CRON[25845]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:17:01 pbx CRON[25914]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 01:25:01 pbx CRON[26180]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:35:01 pbx CRON[26513]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:45:01 pbx CRON[26854]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 01:51:35 pbx systemd[1]: 3CXGatewayService.service: Main process exited, code=killed, status=6/ABRT
Aug 18 01:51:35 pbx systemd[1]: 3CXGatewayService.service: Failed with result 'signal'.
Aug 18 01:51:36 pbx systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Aug 18 01:51:37 pbx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.
Aug 18 01:55:01 pbx CRON[27189]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:05:01 pbx CRON[27518]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:15:01 pbx CRON[27852]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:17:01 pbx CRON[27916]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 02:25:01 pbx CRON[28166]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:35:01 pbx CRON[28452]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:45:01 pbx CRON[28777]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 02:55:01 pbx CRON[29124]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:05:01 pbx CRON[29447]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:15:01 pbx CRON[29768]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:17:01 pbx CRON[29833]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 03:25:01 pbx CRON[30095]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:35:01 pbx CRON[30415]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:45:01 pbx CRON[30768]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 03:55:01 pbx CRON[31159]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:05:01 pbx CRON[31472]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:15:01 pbx CRON[31767]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:17:01 pbx CRON[31828]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 04:25:01 pbx CRON[32077]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:35:01 pbx CRON[32373]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:45:01 pbx CRON[32675]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 04:55:01 pbx CRON[558]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:05:01 pbx CRON[877]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:15:01 pbx CRON[1175]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:17:01 pbx CRON[1237]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 05:25:01 pbx CRON[1485]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:35:01 pbx CRON[1778]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:45:01 pbx CRON[2081]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 05:55:01 pbx CRON[2385]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:05:01 pbx CRON[2687]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:14:28 pbx systemd[1]: Starting Daily apt upgrade and clean activities...
Aug 18 06:14:28 pbx systemd[1]: apt-daily-upgrade.service: Succeeded.
Aug 18 06:14:28 pbx systemd[1]: Started Daily apt upgrade and clean activities.
Aug 18 06:15:01 pbx CRON[3056]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:17:01 pbx CRON[3121]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 06:25:01 pbx CRON[3375]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:25:01 pbx CRON[3376]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Aug 18 06:25:01 pbx cracklib: no dictionary update necessary.
Aug 18 06:25:01 pbx exim4: ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail system possibly broken
Aug 18 06:35:01 pbx CRON[3825]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:45:01 pbx CRON[4131]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 06:55:01 pbx CRON[4443]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 07:05:01 pbx CRON[4752]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 07:15:01 pbx CRON[5066]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 07:17:01 pbx CRON[5125]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 07:18:41 pbx systemd[1]: Created slice User Slice of UID 0.
Aug 18 07:18:41 pbx systemd[1]: Starting User Runtime Directory /run/user/0...
Aug 18 07:18:41 pbx systemd[1]: Started User Runtime Directory /run/user/0.
Aug 18 07:18:41 pbx systemd[1]: Starting User Manager for UID 0...
Aug 18 07:18:42 pbx systemd[5187]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Aug 18 07:18:42 pbx systemd[5187]: Reached target Timers.
Aug 18 07:18:42 pbx systemd[5187]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Aug 18 07:18:42 pbx systemd[5187]: Reached target Paths.
Aug 18 07:18:42 pbx systemd[5187]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Aug 18 07:18:42 pbx systemd[5187]: Listening on GnuPG cryptographic agent and passphrase cache.
Aug 18 07:18:42 pbx systemd[5187]: Listening on GnuPG network certificate management daemon.
Aug 18 07:18:42 pbx systemd[5187]: Starting D-Bus User Message Bus Socket.
Aug 18 07:18:42 pbx systemd[5187]: Listening on D-Bus User Message Bus Socket.
Aug 18 07:18:42 pbx systemd[5187]: Reached target Sockets.
Aug 18 07:18:42 pbx systemd[5187]: Reached target Basic System.
Aug 18 07:18:42 pbx systemd[1]: Started User Manager for UID 0.
Aug 18 07:18:42 pbx systemd[1]: Started Session 70 of user root.
Aug 18 07:18:42 pbx systemd[5187]: Reached target Default.
Aug 18 07:18:42 pbx systemd[5187]: Startup finished in 88ms.
Aug 18 07:19:27 pbx systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 18 07:19:27 pbx systemd[1]: ssh.service: Succeeded.
Aug 18 07:19:27 pbx systemd[1]: Stopped OpenBSD Secure Shell server.
Aug 18 07:19:27 pbx systemd[1]: Starting OpenBSD Secure Shell server...
Aug 18 07:19:27 pbx systemd[1]: Started OpenBSD Secure Shell server.
Aug 18 07:19:58 pbx systemd[1]: Stopping Session 70 of user root.
Aug 18 07:19:58 pbx systemd[1]: Stopped target Graphical Interface.
Aug 18 07:19:58 pbx systemd[1]: Stopped target Multi-User System.
Aug 18 07:19:58 pbx systemd[1]: Stopping 3CX PhoneSystem 01 CallFlow Server...
Aug 18 07:19:58 pbx systemd[1]: Stopping 3CX PhoneSystem Media Server...
Aug 18 07:19:58 pbx systemd[1]: Stopping OpenBSD Secure Shell server...
Aug 18 07:19:58 pbx systemd[1]: Stopping 3CX PhoneSystem 01 System Server...
Aug 18 07:19:58 pbx systemd[1]: Stopping 3CX PhoneSystem 01 IVR Server...
 
Hi @bigdessert,

I don't see any obvious culprit. You mentioned that with the restart it works as intended now? If so, let's monitor this and let me know if it happens again.
 
Thank you for the prompt responses. I will monitor and follow up here if we have the issue again.
 
  • Like
Reactions: VasilisV_3CX
Hello, we are getting the same error randomly after V18 upgrade.
Received today at 12:03
The following service(s) were interrupted:
3CXPhoneSystemMC01
Click to expand...
In syslog i find these messages:
Aug 28 12:03:07 ip-172-26-6-127 systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Aug 28 12:03:07 ip-172-26-6-127 systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.
Aug 28 12:05:01 ip-172-26-6-127 CRON[12724]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Click to expand...
By typing service 3CX* status i get all services running except 3CXPhonenSystemMC01.service:
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Sat 2021-08-28 12:03:07 CEST; 36min ago
Process: 28822 ExecStart=/usr/lib/3cxpbx/3CXManagementConsole --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=kill
Process: 12672 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
Process: 12673 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
Main PID: 28822 (code=killed, signal=ABRT)

Aug 28 05:50:48 ip-172-26-6-127 sudo[31599]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 05:50:48 ip-172-26-6-127 sudo[31599]: pam_unix(sudo:session): session closed for user root
Aug 28 07:50:48 ip-172-26-6-127 sudo[3980]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/sbin/3CXPinningFileUpdate https://downloads-global.3
Aug 28 07:50:48 ip-172-26-6-127 sudo[3980]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 07:50:48 ip-172-26-6-127 sudo[3980]: pam_unix(sudo:session): session closed for user root
Aug 28 09:50:48 ip-172-26-6-127 sudo[8340]: phonesystem : TTY=unknown ; PWD=/var/lib/3cxpbx ; USER=root ; COMMAND=/usr/sbin/3CXPinningFileUpdate https://downloads-global.3
Aug 28 09:50:48 ip-172-26-6-127 sudo[8340]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 28 09:50:48 ip-172-26-6-127 sudo[8340]: pam_unix(sudo:session): session closed for user root
Aug 28 12:03:07 ip-172-26-6-127 systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=killed, status=6/ABRT
Aug 28 12:03:07 ip-172-26-6-127 systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'signal'.
Click to expand...
A "service 3CX* start" fixes it but if i look at CPU usage, when the issue happens CPU usage is capped

1630147761665.png
As you can see it ends at 12:03.
Another thing: on v16 RAM was mostly at 50%, now on v18 at 80%-85%. Is it correct?
1630147407986.png

6 EXT., 1 SBC, AWS Lightsail VM 1 GB RAM, 1 vCPU, 40 GB SSD.
Uptime: 104 days
 
I have not had it crash since my initial post so your issue may be something different.
 
bigdessert said:
I have not had it crash since my initial post so your issue may be something different.
Click to expand...
It’s mostly random. I’m experiencing it always on different PBXs (for now on 14 systems, all 1vCPU, 1GB RAM, 40GB SSD). Only on one has happened twice, and was probably memory related because after upgrade to 2 GB never occurred again. If it happened once, it will for sure happen again.
 
@alberto.azzini

The Hardware Requirements recommended for a hosted 3CX Instance with Amazon EC2/Lightsail do specify machines with 2GB of memory so that indeed might've been the culprit. Do let us know if the issue ever resurfaces now that you've increased it.
 
ChrisC_3CX said:
@alberto.azzini

The Hardware Requirements recommended for a hosted 3CX Instance with Amazon EC2/Lightsail do specify machines with 2GB of memory so that indeed might've been the culprit. Do let us know if the issue ever resurfaces now that you've increased it.
Click to expand...
Yes, i understand and that's why i proceeded in upgrade and imagined that could be the problem.
It just seemed a waste of resources on V.16 since on so tiny PBXs 1GB was enough (500 MB used most of the time). Is V18 more "RAM eater"?
 
  • Like
Reactions: ChrisC_3CX
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,978
Messages
590,100
Members
164,907
Latest member
d.obermeier
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom