No access to Web Management Console since last update

Discussion in '3CX Phone System - General' started by levbe, Feb 13, 2018.

Thread Status:
Not open for further replies.
  1. levbe

    Joined:
    Apr 27, 2008
    Messages:
    33
    Likes Received:
    3
    Since the last update (SP3) on a Debian installation, i can't access the Management Console.
    The web login page says : Server Error.

    When i try to restart the MC service, i get the following error :
    Code:
    Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.
    
    See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
    root@minipc01-3cx:~# systemctl status 3CXPhoneSystemMC01.service
    ● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
       Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
       Active: failed (Result: core-dump) since Tue 2018-02-13 18:59:19 CET; 57s ago
      Process: 962 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
      Process: 960 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
      Process: 935 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, s
    ignal=ILL)
     Main PID: 935 (code=dumped, signal=ILL)
    Does it sounds familiar to someone ?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. levbe

    Joined:
    Apr 27, 2008
    Messages:
    33
    Likes Received:
    3
    In addition to this, i have the following error :
    Code:
    fév 14 10:23:41 minipc01-3cx systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
    
    fév 14 10:23:41 minipc01-3cx systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'core-dump'.
    
    fév 14 10:23:46 minipc01-3cx kernel: iptables[DOS]: IN=enp3s0 OUT= MAC=33:33:00:00:00:01:54:b8:0a:cb:de:20:86:dd SRC=fe80:0000:0000:0000:56b8:0aff:fecb:de20 DST=ff
    
    02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    6,387
    Likes Received:
    455
    Hello @levbe

    Please check your inbox as i have sent you a p.m. with what we will need to troubleshoot your issue
     
  4. emo89

    Joined:
    Feb 14, 2018
    Messages:
    16
    Likes Received:
    1
    Hi, I had the same issue by my system.
    I tried to reinstall 3CX and it works for maybe 30 minutes, after that it logged me out of my system and I can´t login again and get the same Error.
    Restart of MC01 was not possible.
    After I installed 3CX on Windows, because I can´t work without the PBX.
    If you had an resolution, please tell and I want to install it again on my Shuttle PC.
     
  5. levbe

    Joined:
    Apr 27, 2008
    Messages:
    33
    Likes Received:
    3
    The same here...
    I've reinstalled from the ISO image, reconfigured 3cx, got access to the web console and got a "502 bad gateway" when creating extensions.
    3CXPhoneSystemMC01 failed.
    As soon as i get any answer, i re-post it here, of course.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    6,387
    Likes Received:
    455
    Can you share the model of your shuttle pc so we can look into that?
     
  7. emo89

    Joined:
    Feb 14, 2018
    Messages:
    16
    Likes Received:
    1
    Shuttle NC03U with 4gb ram and 120gb ssd
     
  8. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    6,387
    Likes Received:
    455
    @emo89

    Can you please try running the following commend and let me know of the outcome?

    Code:
    sudo -u phonesystem /usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot
     
  9. Ramped

    Joined:
    Jan 4, 2018
    Messages:
    7
    Likes Received:
    1
    Exactly the same thing here after updating earlier today.

    Things worked ok for a while but the console become inaccessible after a while.
    Additionally, We received an email notification;
    "Following service(s) were stopped recently:
    3CXPhoneSystemMC01 (IP Address)"

    When logging into the web interface we now get "Server Error".

    So far have tried the following;
    A) Rebooted the NUC (This did bring things up a couple of times, but nothing now)

    B) Ran the following via SSH: "systemctl start 3CXPhoneSystemMC01"
    Which returned: "Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.
    See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details."

    C) Ran the following via SSH: "systemctl status 3CX*"
    All looked good and returned "active (running)" except for the following:
    3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
    Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
    Active: failed (Result: core-dump) since Fri 2018-02-16 16:11:53 AWST; 1min 46s ago
    Process: 935 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
    Process: 932 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
    Process: 915 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, signal=ILL)
    Main PID: 915 (code=dumped, signal=ILL)

    Feb 16 16:11:50 VOIPServer systemd[1]: Starting 3CX PhoneSystem 01 Management Console...
    Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Main process exited, code=dumped, status=4/ILL
    Feb 16 16:11:53 VOIPServer systemd[1]: Failed to start 3CX PhoneSystem 01 Management Console.
    Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Unit entered failed state.
    Feb 16 16:11:53 VOIPServer systemd[1]: 3CXPhoneSystemMC01.service: Failed with result 'core-dump'.

    D) Out of curiosity, ran the command above as per YiannisH_3CX's comment, which returned: "Illegal instruction"


    3CX Is Debian/Linux Rrunning On-Prem
    Hardware is a "Leader (Australia) NUC i5".

    Phones are still operational, However, access to the Management Console is not available.
    Sorry to Hijack the thread!
     
  10. YiannisH_3CX

    YiannisH_3CX Support Team
    Staff Member 3CX Support

    Joined:
    May 10, 2016
    Messages:
    6,387
    Likes Received:
    455
    @Ramped

    Please check your inbox as i have sent you a p.m. with what we need to further invastigate
     
  11. emo89

    Joined:
    Feb 14, 2018
    Messages:
    16
    Likes Received:
    1
    Hi,
    thanks for the command, but I´ve running 3CX in WIndows for the Moment.
    I wait until the user ramped can tell more if it will run by him when you help him, because the Computer is running in a normal business, I can´t try out at the time, because when it doesn´t work I´ve get problems with my boss.
     
  12. levbe

    Joined:
    Apr 27, 2008
    Messages:
    33
    Likes Received:
    3
    @Ramped : thats exactly my issue also. In my case, it might be a hardware problem but as i can see, you're not using the same hardware than me at all so...

    After discussion with Yiannis, i will try a new install on a different hardware, but if you get your issue solved, please come back here with some informations, it could be useful.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. emo89

    Joined:
    Feb 14, 2018
    Messages:
    16
    Likes Received:
    1
    Hi, I‘ve tried your commend, but the same result like by ramped „illegal instruction“
     
  14. MatthewHood

    Joined:
    Oct 3, 2017
    Messages:
    20
    Likes Received:
    2
    We have the same issue with 3CX Pro on Debian 9 on a VM. we get the same errors...
     
  15. Ramped

    Joined:
    Jan 4, 2018
    Messages:
    7
    Likes Received:
    1
    I have replied to @YiannisH_3CX & will wait for a response. As soon as I have an update, I will pass on any relevant feedback.

    Our phones are all still working, however no Management interface.

    Just so that others can see what I am seeing, I have attached a screen snip of the Managment console login page.
    Capture.JPG Capture.JPG
     
  16. levbe

    Joined:
    Apr 27, 2008
    Messages:
    33
    Likes Received:
    3
    Exactly the same !

    Have you noticed when the error occurs ? Maybe it's not important but in my case, Management Consol fails only when i create new extensions.

    After some tests by Yiannis and R&D team of 3CX on my system (through ssh), they did not found anything, and suspect hardware issue. However, i passed the whole week-end to test hardware (deep memtest, cpu stress tests during hours and hours) and nothing.

    I think 3CX should now really consider this issue as due to SP3 and investigate on their side !

    I already lost one week by testing and waiting, and beeing without a functionnal PBX.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  17. Ramped

    Joined:
    Jan 4, 2018
    Messages:
    7
    Likes Received:
    1
    @levbe

    Funnily enough, I was editing/creating an extension when it stopped working!

    I am still waiting on a reply from @YiannisH_3CX but would agree that if they come up with the same thing, there maybe a bigger issue.. especially considering the system has been working 110% up until the update!
     
  18. emo89

    Joined:
    Feb 14, 2018
    Messages:
    16
    Likes Received:
    1
    I tested it on 2 identical computers (same computer, same memory, same ssd). Both have the same problem.
     
  19. maumee

    Joined:
    Aug 11, 2014
    Messages:
    4
    Likes Received:
    0
    I ma having the exact same issue as noted here. I was editing an extension, and then the nginx / server error issue arose.

    Was installed fresh 2-3 months ago from a 15.5 3CX Distro / ISO. It has been on auto-upgrades every Sunday.

    Also running on Shuttle NC03U with 128GB SSD. I have remote access to this box via web and ssh. I cannot login via the web (because of the noted issue).

    Advice to get this going?
     
  20. manfer

    Joined:
    Oct 19, 2017
    Messages:
    68
    Likes Received:
    8
    Out of curiosity and as I had a test installation on a virtual machine I had decided to start it and test.

    And same result. Getting Server error.

    After digging to find my forgotten credentials, once I was sure I was typing correct credentials and getting "Server error" message I started trying to debug the issue.

    In linux to debug issues you can read logs.

    The first candidate is obviously nginx logs. Curiously access.log is logging nothing at all. I suppose it is not configured. And I got nothing useful from error.log that only was showing an error on checking the letsencrypt certificate, which is expired (long time since last run of the test machine).

    I restarted nginx serveral times just in case it was not running.

    And started looking more logs. There are more logs to inspect. One of them is syslog were I could read:

    Code:
    systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument
    
    that message doesn't sound very good, does it?

    So I stopped nginx just to see /run/nginx.pid disappears correctly. And started it again to see the file is created properly. I read the file to found out it actually contains a PID number and I use ps command to actually confirm it is nginx running with that process ID.

    So, what's wrong. Well, I didn't have a clue at that point but I asked google about that error message on syslog. And found a known bug:
    https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1581864

    So I did the suggested workaround
    Code:
    cd /etc/systemd/system
    mkdir nginx.service.d
    printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" > ./nginx.service.d/override.conf
    systemctl daemon-reload
    
    And 3CX still didn't work but the error on syslog disappeared.

    So decided to restart the machine.

    Management console accesible again.

    Reason of the problem? I suppose it is not really v15.5 SP3 as my test 3CX was not updated, but unattended updates bringing that bug into nginx maybe. Could be possible too, that this issue, though same symptom, is not exactly the same as yours.
     
    #20 manfer, Feb 20, 2018
    Last edited: Feb 20, 2018
    levbe likes this.
Thread Status:
Not open for further replies.