Solved Erroneous Account Lockouts on Management Console

[c.holloway]

Good morning!

I have been having an issue across a variety of clients running both v15.5 and v16 of 3CX. The management console is locking out accounts even when the correct username and password are entered. What's worse, it is locking out after only a single attempt to log in to the server. On one of these instances I tried to alleviate this a bit by whitelisting the address but this just made it impossible to unblock the IP in question as it wasn't blacklisted.

At most of the sites/IPs where this has happened there are no softphones in use and either no phones or no phones that are not properly registered. There are no indications of other failed authentications from the source IP that would account for the lockout after a single attempt (never mind that the correct credentials are being used).

This has happened both with user accounts and the admin account and has occurred in a number of different browers: Chrome, Brave, and Firefox.

Has this happened to anyone else? Any ideas on how to start troubleshooting this?

 

[JohnS_3CX]

Hi @c.holloway

Getting locked out after a single attempt would indicate that more than one attempt has already taken place or that the browser is doing something unexpected (perhaps some browser plugin or something which misbehaves?).

Are we talking about cases where all users connect to the PBX from the same source IP? (ie. from an office that is remote to the PBX)

 

[c.holloway]

John,

It has occurred in cases where all users are remote to the PBX but also in different scenarios. I've had it happen at a site where the PBX is on prem and the IP getting blocked is a single workstation (the user has tried it with Chrome and with Edge).

I've also had it happen at my desk when I'm connecting to an on prem PBX from the outside. I've tested with this particular instance and it occurs with any browser I use, even if there are no plugins installed in the browser. Also, this PBX is where I had the IP whitelisted in the blacklist and entered in the Console Restrictions as an allowed IP. All this seems to have done is gotten my source IP permabanned from the Management Console, I don't seem to have any way to remove the ban. I have since removed the whitelist entry from the blacklist and waited a day or so and tried logging in again but the failed attempt does not create a new blacklist entry for me to remove.

 

[Rhys Haskell]

John,

It has occurred in cases where all users are remote to the PBX but also in different scenarios. I've had it happen at a site where the PBX is on prem and the IP getting blocked is a single workstation (the user has tried it with Chrome and with Edge).

I've also had it happen at my desk when I'm connecting to an on prem PBX from the outside. I've tested with this particular instance and it occurs with any browser I use, even if there are no plugins installed in the browser. Also, this PBX is where I had the IP whitelisted in the blacklist and entered in the Console Restrictions as an allowed IP. All this seems to have done is gotten my source IP permabanned from the Management Console, I don't seem to have any way to remove the ban. I have since removed the whitelist entry from the blacklist and waited a day or so and tried logging in again but the failed attempt does not create a new blacklist entry for me to remove.

Your IP may be in the global IP blacklist if you disable this on your instance does the same issue occur?

 

[c.holloway]

The automatic global 3cx IP Blacklist is not currently turned on. I have found, however, that a couple of the instances I'm currently having issues with are behind on updates. I will apply these tonight and report back tomorrow and let you know if the issue persists.

 

[JohnS_3CX]

Let's break it down:

It has occurred in cases where all users are remote to the PBX
A single person making a mistake will effectively lock out everyone sharing the same WAN IP

it happen at a site where the PBX is on prem and the IP getting blocked is a single workstation
That's normal, if the user made a mistake, their workstation IP should eventually get blocked

I've also had it happen at my desk when I'm connecting to an on prem PBX from the outside
Was it a shared WAN IP you were using? Perhaps you were on a dynamic IP that could have been used elsewhere?

You don't have to answer those questions now, but my point is do not lump them together as a single issue, it will misdirect your investigations. These are all separate cases, always worthy of individual examination.

In answer to your original question, a good idea to start your troubleshooting would be something like the following:

• Who is affected: Single workstation? A whole WAN IP of an office site?
• What do they use: Deskphones only? Clients only? A mix of both?

Based on the above, it's fairly easy to start:

1. Single PC workstation: Can be blocked due to accessing management console, webclient, softclient, or victim of hacking attempt from a compromised PC. Solution: delete saved user/pass on their browser, reprovision their client if they use one, ensure they are not trying to access the management console when they mean to access the webclient instead. Check for possible security breach on their PC.
2. Whole WAN IP: probably users at a single office connected to a remote PBX on a variety of clients or phones. Solution: Whitelist office IP from the very beginning of the deployment. Do not edit a blacklisted IP and change it to allowed. Remove the entry and add it from scratch.
3. Other possible avenues: ensure they have a static WAN IP for cases like 2. Ensure that if they use a failover IP that the 2nd WAN IP is also whitelisted. Ensure they are not trying to access the management console (maybe by mistake or maliciously). You might have to play detective sometimes unfortunately, but finding the one user who triggers the lockout is necessary when all other avenues have been eliminated. Sometimes there is a security breach and there may be hacking attempts from within the office.
4. Console Restrictions: Remember that the management console restriction will still allow you to reach the login screen, but the credentials won't work. This might trick you into thinking that you are blacklisted while there are no entries in the list. Solution: Ensure that this is not what is blocking you by RDP-ing into a local PC to try to login to the local PBX, especially if you are behind a router that rewrites external IP addresses and effectively making the PBX see you coming in from a source IP other than the one you think you are using.
5. Automatic Global 3CX IP Blacklist: Check the 3CX global blacklist option, if enabled and you are connecting from an IP that we have deemed worthy to be blacklisted, you will again be locked out as above. Solution: You can disable this feature temporarily to test if this is the case - we will not reveal the IPs on the PBX (they don't show up in the IP Blacklist) or the reason they were added to the global blacklist so keep this in mind. You need to log in locally into the PBX so have a way to VPN into a remote PBX and appear local to it. For on-site installations, RDP into a PC and connect to the management console locally.
6. Anti-Hacking: Check that you haven't modified the Anti-hacking options under Security, to a point that makes the system super sensitive.

I honestly hope I didn't write too much that my message was lost in the words, and sincerely hope this helps

Happy troubleshooting!

 

[c.holloway]

John,

I realize they are separate issues, I've just put them all in the same thread so as to not be watching several different posts.

Of the three scenarios I have tested two of them recently. The issues with a user at a site remote to the PBX (all phones are on-site) and the issue I have with logging in to an on prem PBX remotely in which a single attempt to log in delivers a message that the IP has been blacklisted but no corresponding blacklist entry shows up.

To the first case: user at remote site having trouble getting logged in. I was able to determine that it was indeed one of the scenarios you presented, he was trying to log in to the management console rather than the webclient. He was changing profiles in Chrome, from one where he had a bookmark and a saved password. This profile was always working and didn't have issues. I didn't even think to ask him this as I wrongly assumed he had just copied the URL from his bookmark to the new profile. So AFAIK this issue is resolved.

The second case: I cannot log in to an on-prem instance remotely. I continue to have these issues; I fully updated the instance last night. Everywhere that I have tested from has a single static WAN IP. I input this IP into the Console Restrictions page then attempt to log in from this IP. The login page comes up properly but any attempt to log in is met with the error "Login access denied. Too many incorrect login attempts. Try later or contact the Administrator. " after a single attempt using the admin credentials. This IP is not added into the blacklist, I imagine because it is whitelisted from the Console Restrictions page. If I disable the console restrictions and allow logins from any IP address the issue goes away. I'm currently trying to confirm that the IP hitting the nginx server is indeed the same as is set in the Console Restriction page (to rule out NAT as an issue) but i'm running into an issue of the access.log file being blank. This instance is running on a Debian 9 VM.


With regards to the other scenario, where an on prem user is getting locked out of the management console repeatedly despite being sure the credentials are correct, I haven't done extensive troubleshooting with the user and therefor can't confirm if it is user error or not. We've pretty frequently had to go in and reset the password for him to get him logged in but hasn't reached out about it in a while though this may be due to frustration on his part. It's entirely possible the user is just mistyping or having some other issue. I have had him remove the saved password from his browser when he's reported being locked out on the first attempt and he still reports the same thing. However, due to my uncertainty about this user's reliableness I'm not overly concerned with this particular scenario.

 

[c.holloway]

Update:

I found that access logging was turned off under /etc/nginx/sites-enable/3cxpbx. I got that turned back on and got some logs. It turns out there is some kind of NAT problem as the access logs show the access request coming from the remote sites WAN IP rather than my offices WAN IP

 

[c.holloway]

I found the issue. It was a problem with how the NAT rule was set up on their Barracuda firewall. TL;DR the wrong source IP was being delivered to 3cx which was rejecting the connection attempts as the IP being delivered was not in the Console Restrictions page

You can go ahead and close this out and thanks John for your help!

 

[JohnS_3CX]

Hi @c.holloway

Glad to help and really happy to hear it was all sorted!