• V20: 3CX Re-engineered. Get V20 for increased security, better call management, a new admin console and Windows softphone. Learn More.

[INFO] Server Manager error

Status
Not open for further replies.

Marari

Silver Partner
Advanced Certified
Joined
Sep 16, 2007
Messages
340
Reaction score
90
Running 3CX Professional 15.5.13103.5
(so are all other instances mentioned in this post)

Within 24 hours, all of my google cloud hosted 3CX installations and local 3CX installations have had a server manager failure. Outbound calls were working fine, but all inbound calls received a busy signal.

A reboot of each of the servers has corrected the issue, and a review of the activity logs has shown a massive spike in entries that look like this:

Code:
08/12/2018 8:35:06 AM - [CM102001]: Authentication failed for AuthFail Recv Req REGISTER from X.X.X.X:YYYY tid=547fa98f-f3e6-4677-a76f-a6f5fd16b345 Call-ID=hwpoycmaaybrrcpdvfxpblyvrywoaomceddtytacpooyemhxjb:
REGISTER sip:Z.Z.Z.Z SIP/2.0
Via: SIP/2.0/UDP X.X.X.X:YYYY;branch=z9hG4bK547fa98f-f3e6-4677-a76f-a6f5fd16b345;rport=YYYY
Max-Forwards: 70
Contact: <sip:[email protected]:YYYY;rinstance=8ecd086809658d41>
To: "5216"<sip:[email protected]>
From: "5216"<sip:[email protected]>;tag=ltkqsuxt
Call-ID: hwpoycmaaybrrcpdvfxpblyvrywoaomceddtytacpooyemhxjb
CSeq: 2 REGISTER
Expires: 3600
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, SUBSCRIBE, NOTIFY, REFER, INFO, MESSAGE
Proxy-Authorization: Digest username="5216",realm="3CXPhoneSystem",nonce="414d53595b70378933:706cb4cc51e959f334362e708d629d1a",response="abfc9f09f6efc6940709399a39d93786",uri="sip:Z.Z.Z.Z",algorithm=MD5
Supported: 100rel
User-Agent: 3CXPhoneSystem
Content-Length: 0

 ; Reason: Credentials don't match, check that authorization-ID and password match the ones in extension settings

X.X.X.X is the "offending" IP address
YYYY is the port that the offending IP address is using
Z.Z.Z.Z is the IP address of the 3CX server

You can replace the X.X.X.X and YYYY with multiple sources and ports. There are quite literally dozens of attempts every hour.

I've had a field day adding complete ranges to the IP blacklist to reduce the number of attacks as these don't seem to trigger auto-blocking. Or if they do, it's a rare instance in comparison to the number of activity log entries.
 
Same thing here affected a number of PBX's.
 
Interesting find.

We also had one instance produce this same behaviour today, and a restart of the VPS corrected it.

I'll have to look at the logs and see if we're seeing the same traffic.
 
We've had the same issue today on multiple new PBX's on this version. It's only been the ones we've deployed in the last week or two though, Other PBX's which have running for ages and updated to this version seem to have been fine.
 
Are all of the affected instances running Debian? If so then please note that the reason of the reboot over the weekend is that Debian published a security update for postgress as you can see from here https://www.debian.org/security/2018/dsa-4269 which the debian installation applied automatically and that is what caused your services to restart. However do not worry about this as in the final SP6 release we will take care of this so it doesn't happen again.
 
Are all of the affected instances running Debian? If so then please note that the reason of the reboot over the weekend is that Debian published a security update for postgress as you can see from here https://www.debian.org/security/2018/dsa-4269 which the debian installation applied automatically and that is what caused your services to restart. However do not worry about this as in the final SP6 release we will take care of this so it doesn't happen again.

Yes, all instances are running Debian.
 
Same all instances on Debian and new installs.
 
I get those entries all day every day - on a windows server.
My firewall block list is huge - but not all encompassing. There’s always some that slip thru.
We had a total power outage over weekend. Batteries didn’t last long enough. When 3CX machine booted nginx front end wasn’t working. Database was up - calls were processing. But no presence or management. Had to start nginx manually.

Not sure if related at all. Or maybe I should make nginx a delayed start on windows services.
 
I get those entries all day every day - on a windows server.
My firewall block list is huge - but not all encompassing. There’s always some that slip thru.
We had a total power outage over weekend. Batteries didn’t last long enough. When 3CX machine booted nginx front end wasn’t working. Database was up - calls were processing. But no presence or management. Had to start nginx manually.

Not sure if related at all. Or maybe I should make nginx a delayed start on windows services.

I'm almost tempted to block all except the local LAN, WAN address, NOC address, and the VOIP providers in use.

I know that limits the use of the mobile client, but that's really a non-issue at the moment and has been easily replaced with "simultaneously ring mobile".
 
I had the reboot also during the weekend. My 3CX is Debian based also.
 
Same thing here affected a number of PBX's.
We have been seeing a lot of the same across windows and debian platform...same IP hits all our 3CX systems all around the same time. Started on 8/10/18 and is still going!
Running 3CX Professional 15.5.13103.5
(so are all other instances mentioned in this post)

Within 24 hours, all of my google cloud hosted 3CX installations and local 3CX installations have had a server manager failure. Outbound calls were working fine, but all inbound calls received a busy signal.

A reboot of each of the servers has corrected the issue, and a review of the activity logs has shown a massive spike in entries that look like this:

Code:
08/12/2018 8:35:06 AM - [CM102001]: Authentication failed for AuthFail Recv Req REGISTER from X.X.X.X:YYYY tid=547fa98f-f3e6-4677-a76f-a6f5fd16b345 Call-ID=hwpoycmaaybrrcpdvfxpblyvrywoaomceddtytacpooyemhxjb:
REGISTER sip:Z.Z.Z.Z SIP/2.0
Via: SIP/2.0/UDP X.X.X.X:YYYY;branch=z9hG4bK547fa98f-f3e6-4677-a76f-a6f5fd16b345;rport=YYYY
Max-Forwards: 70
Contact: <sip:[email protected]:YYYY;rinstance=8ecd086809658d41>
To: "5216"<sip:[email protected]>
From: "5216"<sip:[email protected]>;tag=ltkqsuxt
Call-ID: hwpoycmaaybrrcpdvfxpblyvrywoaomceddtytacpooyemhxjb
CSeq: 2 REGISTER
Expires: 3600
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, SUBSCRIBE, NOTIFY, REFER, INFO, MESSAGE
Proxy-Authorization: Digest username="5216",realm="3CXPhoneSystem",nonce="414d53595b70378933:706cb4cc51e959f334362e708d629d1a",response="abfc9f09f6efc6940709399a39d93786",uri="sip:Z.Z.Z.Z",algorithm=MD5
Supported: 100rel
User-Agent: 3CXPhoneSystem
Content-Length: 0

 ; Reason: Credentials don't match, check that authorization-ID and password match the ones in extension settings

X.X.X.X is the "offending" IP address
YYYY is the port that the offending IP address is using
Z.Z.Z.Z is the IP address of the 3CX server

You can replace the X.X.X.X and YYYY with multiple sources and ports. There are quite literally dozens of attempts every hour.

I've had a field day adding complete ranges to the IP blacklist to reduce the number of attacks as these don't seem to trigger auto-blocking. Or if they do, it's a rare instance in comparison to the number of activity log entries.
 
Had the same issue today on a Windows and Debian system. Seems like there have been a big attempt to hack 3CX last week.
 
Are all of the affected instances running Debian? If so then please note that the reason of the reboot over the weekend is that Debian published a security update for postgress as you can see from here https://www.debian.org/security/2018/dsa-4269 which the debian installation applied automatically and that is what caused your services to restart. However do not worry about this as in the final SP6 release we will take care of this so it doesn't happen again.

That explains why most of our Debian installs restarted, thanks for the update :)

It doesn't explain the recent services failing/stopping and having to manually reboot the Debian instance though.
 
We just had another Debian instance do the same thing. (Unable to login, 'server error', unable to receive calls)

I've logged in and run
Code:
systemctl status 3CX*

and found only one service failed;

Code:
â 3CXPhoneSystem01.service - 3CX PhoneSystem 01 SIP Server
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystem01.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Tue 2018-08-14 11:01:59 ACST; 1 day 22h ago
  Process: 12187 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystem01.pid (code=exited, status=0/SUCCESS)
 Main PID: 509 (code=killed, signal=SEGV)

I then ran

Code:
/usr/sbin/3CXStartServices

And it's happy again. (Can now login, as soon as the services started the trunk re-registered and calls are happy).

Not sure what to look for next.
 
@pact

I would recommend creating a ticket if the issue persists as our support department will need to go through the PBX logs to determine the cause of the issue.
 
@pact

I would recommend creating a ticket if the issue persists as our support department will need to go through the PBX logs to determine the cause of the issue.

If we observe a third occurrence then I will lodge a ticket :) Third times the charm they say.
 
Status
Not open for further replies.
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.