Getting back into my Amazon S3 system and protecting it

[swiftechinc]

I really don't understand what's going on here. Firewall is set to block IPs for failed logins. I have access to six different networks to attempt logging in from. Two are whitelisted.
But I can't get in from any of them. They all say I'm locked out for too many failed login attempts no matter where I access from and it's been 10 days.
All of the extensions and everything work fine. I don't "need" in it right now. But I'd rather fix this now than when I do need in.

Part 1.

3CX backs up regularly and I have a backup downloaded from the 3cx console from two weeks ago that I could restore to IF I could get in. I don't know if there's a way to trigger it though without being logged in. I can access the CLI side of the server itself but I'm not sure that this will do me any good.

I have a snapshot from two weeks ago but the restore process seems a bit convoluted. I have an AMI from nearly a year ago but that's opening a different can of worms.
I did shut it down for 5 minutes thinking maybe it was rebooting too fast. I had hoped that the IP block list would either purge or at least leave a few minutes before implementing automatically applied firewall rules. I do see a lot of similar posts about this and that's something 3CX might want to consider adding in a future build.... Maybe a user selectable setting for a period of time after reboot to allow traffic from blocked IPs? Heck, 30 seconds would be enough. lol
I'm still baffled at how the 3 day lockout is remaining persistent for 10 days now.

Another option I was considering although I haven't dug in to see if it might work, would be to attach a second network adapter to the virtual machine. I just have no idea at the moment whether this would be useful or not. I don't think it would work but I'm trying to think of something outside the box. I can't swap the adapter without breaking the system. It sure would be nice if there was some kind of alternate way in. Maybe building a backup route into the system or something. I don't know. Just annoyed. And I simply don't understand why after 10 days I still can't get back in on any of these IPs which are all different ranges.

Part 2.
Somehow I'll figure this out even if I have to restore using the snapshot which assuming no one has a better idea, I'll be doing this weekend. But following that, I need to find some way to prevent this going forward. The recommended settings are already in place regarding the various security tools like login attempts, frequency, etc. Logins come in over a standard port, 5001. Would I be able to switch this to some other random port without breaking anything? Would that maybe prevent or at least put a stop to this? Any other suggestions?

 

[SweetAction]

So anything in RFC1918 is always in the console restrictions whitelist, which sounds like the actual block here if 6 networks are all blocked on first attempt. Not failed login attempt blacklist.

So can you get a machine in the same network as 3CX, then visit 3CX management page by IP and see if you can login?

 

[swiftechinc]

It's an amazon cloud server. I can't get to it without using the public IP from a public IP.

 

[SweetAction]

It's an amazon cloud server. I can't get to it without using the public IP from a public IP.

There are no other servers in the same VPC? Can you spin something up quickly?

 

[swiftechinc]

I've thought about that but since I don't have anything else on there except the 3cx system, the time would be more than restoring from the backup. Then again, I really want to see the logs which I'll lose if I do just restore it. So I think I'll keep trying. Maybe I'll do that. I have to think it through. Maybe a cloud router, move the IP to the router, give an internal range and put a VPC on the same router. Sounds fun. lol

Before anyone asks, this isn't a bad password issue. I can get into my extension via webclient just fine and so can everyone els
I tried my personal extension from here and it's locked as well. But of course the web app allowed me to login fine......But it's located at www.yyyy.zzz:5001/webclient

So the web app and the admin portal are in separate locations. And being down a path allows it to evade these types of attacks. So another possible security solution...moving the whole admin portal to a subfolder. For the webportal, the location is https://my.3cxsystem.url:5001/webclient. Since I've NEVER had an issue with attacks on that and https://my.3cxsystem.url:5001 leaves me dealing with this kind of issue continually, it would be great if 3cx would look into doing this.

What I can't figure out is, well, take this morning for instance... I left town and drove to a town 3 1/2 hours southwest in another state overnight. While I was driving, 3cx pushed an update out to the system. I'm visiting family so I jumped onto my laptop and did a few things...1) selected an option for a random MAC (It would be great to limit access via MAC), downloaded Edge Canary, (already running beta and Dev). connected to the internet via my nephew's wifi which is Comcast - someone not even in my region. Bam. First time. "Login Access Denied. Too many incorrect login attempts. Try later or contact the Administrator."......which would be me.

By now the time spent on an alternative has already exceeded the time it will probably take me to restore it. But I need to do that on a weekend so I may as well poke and prod at it and see what I can learn from it in the meantime. lol

So the lockout is either account-based, or a huge chunk, if not all of the public IP space has been blocked. I don't have a backup admin account. I'm not sure it would be relevant because I can't access any web portals at all. No one can. So I'm leaning to the

IP blocks that don't make sense to me but maybe I'm missing something. But that wouldn't matter if the lockout is the entire internet. If it is though, why does it not let me in 10 days later? If it were the account bad password block alone then it should cure in 30 minutes.

I'm assuming the IP block is ahead of the login auth check so blocked IPs aren't likely extending the lockout time by continually hitting the server while locked out. But why are they not expiring after 3 days? I'm 10 days in.

Still would like to know if changing the port to whatever we want it to be might either be possible going forward or if not, if 3cx will consider allowing it. Would be at least nice to put a backup auth path in. Port 19774 works for me. lol

Well, time to put the toys away and get some work done. I'll come back to this later as I spend the day considering whether or not call flow designer might be able to help me. lol

 

[SweetAction]

Yeah, almost for sure the issue is console restrictions are enabled. Like 99% sure - this is the exact symptoms (you can load the page, first login says wrong credentials no matter what you enter).

Hitting the PBX from an address in RFC1918 which is in the whitelist and will let you get back in.

 

[swiftechinc]

I'm trying to sort this out in my head. The console restrictions makes sense except I never enabled that. I've been able to remote in for quite some time and haven't changed anything in security for quite a while.

I can reinstall it and then load a backup which would be pretty quick I think but I really would like to get the logs and see what caused this.

I've considered spinning up a vpc but I'm not entirely sure how to go about it in a way that would allow me to access it. Would a simple windows install on the same partition generate an internal IP within the same range? If so that may be the best way to go as a hacker would then have to come in that way as well which would just be a pain in the rear.

One other thought.. if console restrictions are enabled then why am I getting emails 2-3 times a day with new blacklisted IPs. What's the point in blacklisting IP addresses if it simply can't be accessed from outside?

 

[SweetAction]

I'm not going to spend a ton of time here discussing here AWS networking works, but I'll lay out the basics
1. Assuming you are using EC2 Instance for your 3CX VM, you configured a subnet for the machine.
2. If you make another EC2 instance and choose the same subnet, you will be able to access the 3CX VM by it's internal IP (10.x.x.x or 192.168.x.x or 172.16.x.x/12)

If you have SSH access to the 3CX machine, you could do

 ssh 'your-username' @ 'your-server-ip' -L 5001:localhost:5001

Then open webpage, Type https://localhost:5001 login and go check it out.

 

[swiftechinc]

I'm not going to spend a ton of time here discussing here AWS networking works, but I'll lay out the basics
1. Assuming you are using EC2 Instance for your 3CX VM, you configured a subnet for the machine.
2. If you make another EC2 instance and choose the same subnet, you will be able to access the 3CX VM by it's internal IP (10.x.x.x or 192.168.x.x or 172.16.x.x/12)

If you have SSH access to the 3CX machine, you could do

 ssh 'your-username' @ 'your-server-ip' -L 5001:localhost:5001

Then open webpage, Type https://localhost:5001 login and go check it out.

Fantastic! Thanks!
It is on EC2 and I should have SSH access but something isn't right with it as it won't connect. I had this problem last year and fixed it but I can't remember what it was right now. Probably a loose stupidizer. I'll figure it out.

BUT here's the good news. I was browsing through backup files and found the problem!
Short history... In Feb there was an update that kept failing. I couldn't find the problem so I rolled back 1 month at a time until I landed on a backup that allowed the update. Once the update succeeded I backed up again manually. Then the autobackup ran two days later.
So I looked at the *.xml files from several; dates and here's what I found....
In the The 2/16 backup made before I started applying older backups, the IP_WHITELIST is ON and there are several IP addresses as well as a couple /22 blocks which were there so I could access from my phone when I'm out if needed.
The 2/18 backup was made after the restore and update. It shows IP_WHITELIST as OFF and no IPs are listed. This makes sense as the backup that ended up working was made before I implemented the Whitelist. I do wonder if the Whitelist itself caused the problem with the backup although I'm unsure as to why that would be.
The backup from 2/20 was an automated backup and the IP Whitelist was ON again. But the whitelist has a single /16 range listed and it's not one from my list from before.
I didn't remember turning it back on and I didn't recognize the IP. I looked it up and it's an IP of a local carrier here in town but not one that I've used.
I started looking through our trunk-provider bill and found that IP was one that our former engineer had at his home where he worked on the system for us until Alzheimer's got bad enough that I had to ever so gently convince him that he's done enough and now it's time for him to relax with his family. This was last fall.
So, it appears that he regularly attempts to login to the system and when I restored the older backup his old password became valid again which allowed him access to the system. When he was able to login he saw that the whitelist wasn't enabled so he added his IP and re-enabled it which then locked out anything but IPs from his provider.
So I'm glad that's sorted out. He doesn't remember doing this and I'm not going to give him a hard time about it. If I can't figure out how to fix my ssh access I'll probably just edit the xml file on the latest backup, replacing his IP with mine, reinstall, and then apply that backup.
Thanks for all of the input. This case is closed.