Please help, 3CX Alert: Trunk Failover or max amount of calls available for the trunk has been reached. Possible compromised system.

[Youdjin]

Hello experts,
today was one of those "Oh boy" days.
This morning around 8:40am i started receiving Alerts from my system about the trunk limits. Continuously and they would not stop. However there's more to that. One of my user or the whole department started receiving calls via one of the trunks about posted shipments from one of my user's account. Users themselves did not post anything and they did not know what is going on. Well one of the user's brokerage account has been compromised reaching hundreds of posts on the board. What's interesting is the shipments posts included our sip trunk number with my user's extension. At the same time my 3cx system started receiving all these alerts about trunk limits every minute. When i opened the Active Calls window and call logs i saw there tens or hundreds of calls from random numbers made to our sip trunk. In order to mitigate this attack and somehow release the load so that my other sip trunks can make calls, i immediately disabled the attacked sip trunk, unchecked Inbound and Outbound calls under sip trunk Options and disabled the actual sip Trunk from Telnyx dashboard itself. Yes, this trunk was originally bought from Telnyx. We immediately reset User's email, passwords, extension passwords.
The system immediately started working and back to normal again however here's the interesting part:
Right now its almost 3pm and the alerts are still coming about maximum calls on the trunk, every minute. My inbox is full of them. What i receive in the email is this as an example:

Device 4072057923@(Ln.10003@DL) had no available outgoing trunk(s) to call(31784)
Device 6305800777@(Ln.10003@DL) had no available outgoing trunk(s) to call(31847)
Device 2165700978@(Ln.10003@DL) had no available outgoing trunk(s) to call(31848)

A trunk failover occurs when the maximum amount of calls available via the trunk have been exceeded.

These device numbers are random. This is where i need your help. My question is: Does this look like my system is compromised? Is there an infected internal device that is still trying to make calls via this trunk? What can i do to stop this? I removed one of the generic operator extensions that was in this department as a precaution and looks like right around that time the alerts have stopped. I unplugged some of the desk phones for now from that department. Only 4 people in that department. What's throwing me off is that
no other notifications from other trunks are coming, and the system is working as normal meaning my users can make and receive calls, otherwise the whole system would have gone rogue.

 

[BrunoI_3CX]

It sounds like you have taken the correct first steps by disabling the trunk at the provider level.

Which is the current status of the issue? You are still receiving alerts? As a first step, our suggestion is to check which users are trying to trigger those calls and revoke it's credentials.

Additionally, you should check whether any email account might have been compromised in order to get access to the PBX.

Please find the following resources to audit your security configuration:
- Don’t be “THAT” Guy Vol.1: Keep Complex Credentials
- Don’t be “THAT” Guy Vol.2: Call Fraud
- Don’t be “THAT” Guy Vol.3: Top 4 PBX Security Tips
- Don’t be “THAT” Guy Vol.4: Monitor Your Instance
- Common Mistakes Which Lead to Call Fraud

 

[Youdjin]

Hi Bruno,
thanks for reply. Alerts have stopped.
That's the thing that's confusing to me, no user extension is trying to initiate the calls. I checked the logs. The alerts show devices that are trying to initiate, however these devices are unknown.
The generic Operator extension that i disabled, was created long time ago without an email address. Way back in the days of V18. Apparently disabling or deleting it, has stopped this whole mess.
I will check on those resources.

 

[VoIPTools]

Many, many years ago we had a customer's employee respond to the "your computer has a virus" scam and installed the anti-virus "fix". The software the employee installed scanned their network, learned what they needed through various key loggers, etc., and setup their own surreptitious SIP clients that began making calls through their PBX to foreign countries. Your situation sounds very similar. My suggestion is that you perform an urgent company-wide update of your anti-virus definitions and start scanning every system now including laptops and home computers. You can change passwords, but that does not help if they are watching you change your password.

 

[SteveITS]

FYI resetting the web client password on the login page doesn’t disconnect apps. Reset button in the user list does. Also check open sessions in the admin panel.