3CX Console impossible to log in

[AWS2P]

Hi ,
today a new customer came to me with a V16 Debian running pbx on a google cloud instance.
This pbx since last update has lost his ability to let admin user open a console session.

URL on https://domain.3cx.fr reach 3CX web login page, and when you try to use login/ password you get directly a msg saying connection denied, too much incorrect tries (even it's the first time you try).


This is not a login / password problem , i got an access with SSH and gone to backup folder and got a zipped backup file, opened it and checked console login & password , they are the good ones.

We tried from 3 differents locations and got each time same message directly on the first try.

PBX initial install on Google cloud was done with pbxexpress and same login password was working since 2 years.

I'm not an expert on linux so I dont know what i could do from SSH to help in troubleshouting or fix this problem.

Do you think this is something i can fix without a complete reinstall ?

 

[cobaltit]

I'm guessing someone made a mistake in the admin console restrictions section. I know @BrenttG can fix this via SSH.

 

[AWS2P]

Normally customer said to me nothing has been changed in console restrictions

They have no one in the company knowing something about 3CX

 

[jakorsme]

Sometimes (not always) after a reboot of the VM, I can get into the admin console again. However after an hour or so CPU is pegged on the vm at 100%. But way before that admin access is gone.

Brand new install that went fine and then I went to bed. Starting the next day this nightmare started.

 

[AWS2P]

reboot already tried twice with no success for me

 

[cobaltit]

Well assuming it is an IP restriction the other option is to use SSH port forwarding to try and access the console. I don't think localhost is ever blocked.

 

[BrenttG]

If you send me the SSH details ill help you by taking a look at it, and seeing if i can tell why the cpu goes into the weeds.

 

[jakorsme]

I think it has something to do with the iptable rules that 3CX creates during installation on the VM debian instance. It's like they got everything so tight the mgt console won't even allow connections.

This is what I did from the debian vm console.

1) tried wget on both localhost and internal lan ip on port 5001 - I got back the management console html just fine, rather than the usual can't connect - now I'm thinking the firewall on this debian 9 vm.

2) flushed all the iptable rules
3) did a: iptables -I INPUT -j ACCEPT

4) went to my browser on another internal computer and went to the 3CX debian vm: ie. it is https://192.168.1.183:5001

Low and behold the Management console login screen popped right up.

5) I entered my credentials and clicked login.

Nothing - like I didn't click the login button at all.

6) clicked on refresh.

Magic - I'm now in the Management Console.

This is a mess. There's problems with the 3CX constructed iptable rules applied during install.

Still don't understand the sudden spike to 100% CPU after a few hours though.

 

[BrenttG]

the iptables rules created on a new debian 9 based PBX are formed correctly, i have reviewed them numerous times as we have over 100 debian 3cx servers we host.

It is possible something has caused the ruleset to become corrupted, if you installed any addon packages they also could have altered the ruleset, or if someone tried to change any of them with the shell, this too could have forked them.

Did you install from the 3CX ISO, or did you do what we call a "Wing It" install where you downloaded debian 9 or 10, and then installed the packages manually, and chopped it all togeather yourself???

 

[jakorsme]

Installed directly from your iso - no other additions or modifications, other than initial configuration through the 3CX Management console. Nothing else on this vm.

Also when I removed your rules as stated above, the firewall checker started succeeding where before it was not finding various stun servers and other failures.

Our main firewall that sits between our internal lan and the internet has the inbound nat rules for 3CX and works fine.

 

[BrenttG]

From my ISO?? My Rules??

Somehow your VM got messed up, we have never seen this except when someone has played with things... and then come to us for help, which has happened a few times.

What virtualization platform is it on, and what virtual NIC type did you use?

 

[jakorsme]

Well not yours personally - the 3cx iso and the 3cx install generated rules.

I can't speak to what others have done, but I simply installed and basically configured through the 3cx, backed it up through the 3cx mgt console, then went to bed.
Got up the next morning with everything is going wrong. No one touched this overnight as I'm the only one with access.

It's on vmware 6.5 and is on the same physical nic as other vms, as well as what our 3cx system was on functioning for two years before upgrading to debian 9 and 3cx blowing itself up. that's why the install from scratch.

 

[BrenttG]

No, not the physical nic, what virtual nic type did you use, E1000, vmxnet3, etc?

 

[jakorsme]

VMXNET3

 

[BrenttG]

not sure that its the issue, but we build all of ours using the E1000's as they rely on intel based drivers, and are known for being more compatible with things, It is possible the vmxnet3 driver is going to the weeds and that could also be related to the cpu spikes.

If you want to try something, do a full backup, and reinstall a new VM with an E1000 nic, and restore to it, and see if the problems vanish....

When you are reloading iptables, and other stuff, that could also be causing the interface to reset which would have the effect of making it seem like iptables was to blame, even tho it was the interface reset during commiting your changes that actually made the impact.

A lot of people swear by vmxnet3, and I am one of them for most uses, however I have also run into situations where they cause very strange issues, usually related to driver problems. Depending on what NICs you have on the actual hardware, i have heard there can also sometimes be issues in which the virtual vmxnet3 nics have quirky issues with some hardware nics on the hosts, but i have not personally witnessed that one myself, so far... But we also run Intel X710 10gb nics and i think its in the realm of reasonable to think that there could be a correlation to the hardware nics and virtual nics being similar families.


fair to say we have just a lil bit of vmware/3cx cooking

 

[jakorsme]

mmmm. It is all definitely consistent with a flakey interface, but that's why I mentioned that other VMs are virtualizing off the same h/w nic (also vmxnet3) without issue. However, and it's a potentially big however. We don't use Debian normally (only 3CX uses it), as we are all Centos-based with a Ubuntu or two.

However, also we did have 3CX running for a couple years on this vmware server with Debian 8 - but it was the pre-6 3CX.

I think changing the vnic type on a new vm install is worth a shot. Hopefully the console will come up for long enough to do a backup, so I don't have to revert to the backup I took at the end of the basic 3CX config.

 

[cobaltit]

You can do a command line backup, no console needed

https://www.3cx.com/docs/backup-restore-command-line/

 

[jakorsme]

Well, tried the new VM using the E1000 vnic instead.
1) Installed from the iso and when it had installed and got to the configuration choice of browser vs. shell prompt, I chose browser as always - but I couldn't even connect on 5015 this time.

2) so I rebooted and chose configure from shell prompt. That worked and when it completed I was able to login to the management console on 5001.

3) I restored my backed up configuration, which went fine.

4) I tried an inbound phone call and it went just fine and produced a voicemail.

5) went away for a couple hours.

6) tried browsing to the 5001 mgr console - connection refused again.

7) The console for the vm is also frozen again like before.

8) Inbound calls now fail.

Apparrently the switch to E1000 vnic did nothing.

A couple reminders.

1) we had the previous 3cx version running for two years just fine with our perimeter firewall which hasn't changed.
2) Our other VMs are unaffected which pretty much eliminates hardware network as the issue.
3 Let 3cx config the VM's firewall - ie. I didn't mess with the rules.

 

[BrenttG]

something in your scenario is performing strangely, because we deploy about 2-3 of these a month using their ISO....

Did you load the vmware tools, or did you leave the system alone? what did you do here? Or did you load the open source tools?

 

[jakorsme]

On the previous VM I installed the vmware tools. On this most recent VM with the E1000 vnic I did not install them, nor did I do anything beyond the 3CX install & cmd line config.