Specific Ring Group Stops Ringing at random times and shows missed calls

[JohanThiart]

We have 3 ring groups setup on 3CX Enterprise on an AWS Lightsail server. Each group has 2 users and one of the groups has 5 users/phones. All phones are Xiaomi Redmi 9A Smartphones running Android 10 with the 3CX app. The ring groups with the 2 users works flawlessly without issues. The 3cx system worked fine for all groups for the past 3 months. Since around 2 weeks ago (Not sure if a 3CX Server update was done automatically) the ring group with 5 users/phones started to stop ringing around midday 12h00. There are nothing changed on the 3cx app except we have tried the "keep Alive" setting to be turned on for troubleshooting purposes, rebooted phones multiple times etc to try to sort the issue. The ring group with the issue just shows missed calls on the 3cx app but does not ring. After a while or when phones rebooted it works again but will stop the next day around midday 12h00 again. All phones on all ring groups are the same version Android 10 and 3cx newest App version. Please help, I have tried checking phones and server but since 2 of the ring groups work fine I cannot see that it is a 3cx server issue and rather a ring group or 3cx app issue.

 

[ChrisC_3CX]

We might have to gather a bit more information on this first so:

• What exact version of the 3CX PBX and what version of the 3CX Android Apps are you currently running?

• Are the 5 affected devices on the same network?

• If yes, have you tried any on a different one while the issue is present or even switching them to a Data connection(LTE)?

• Does absolutely anything else happen at 12:00. Scheduled reports, Office hours change or break times applied or anything else you can think of?

• When you tried the "Keep Alive" setting, did it actually fix the problem?

• When you receive the missed call on the android devices, is it a PUSH notification or do you only get it in the "3CX App >> Recents section"?

 

[JohanThiart]

Thanks please see below, Let's call it Ring group 1,2 and 3 where 3 is the problem ring group for answers below

1.) 3CX Server - 18.0 (Build 314), 3CX Apps on all Xiaomi Redmi 9A Phones 18.0.8.86

2.) Ring group 2 and 3 on same Wifi, Ring Group 2 working, Ring Group 3 is the problem Ring Group, Ring Group 1 on another network and working

3.) Yes, external 4G Data connections working to 3CX Server, cannot test this unfortunately for Ring Group 3 which has a problem since no sim cards installed on the phones and all working via existing Wifi Networks, Wifi Network is on 25Mbps Up / 25 Mbps down and utilization around 10% average on network. Phones are provisioned on the Network adapter IP on users on 3CX Server and not on domain name (Also tried domain name but problem still remained).

3.) No Schedules or break times, Office hours setup from 7h45 to 16h45

4.) It seems NO Push notifications are received. All Missed calls are from Recents list on 3cx app manually checked. Keep Alive are set each morning but it seems after a few hours it is not active anymore. Ring Group 1 and Ring group 2 which is working advised they click the Keep Alive setting several times during the day to ensure it is active again. Ring Group 3 which has a problem only chooses it once every morning. It seems overall push notifications is a problem on all groups.

5.) See above, missed calls manually checked on 3CX App on phones.

 

[ChrisC_3CX]

If it works when you enable "Keep Alive" then it almost definitely is an issue with PUSH.

First thing I'd recommend is verifying that your network and device connectivity satisfies Google's FCM Requirements. What you can also do, is take a look at the settings and make sure that your Xiaomi devices are configured this way to optimize them for 3CX

If you want, to check 3CX's end, you can:

1. Enable Verbose Logging from within "Management Console >> Dashboard >> Activity Log >> Settings".
2. Make the same call and confirm you had the issue.
3. From the "Management Console >> Dashboard >> Activity Log" section go to "Logs >> Instance" and download the 3cxSystemService.log.

Search through it and go to the time where you made the call, you should see something like this:

|Info|Sending PUSH to extension EXT# device DEVICE_NAME
|Info|Length of PUSH data to be sent to FCM: 504 bytes
|Info|Response from FCM: Status: OK, Reason: OK

The last line is what confirms that the 3CX sent the PUSH and Google confirmed they received it. Do you see this for all phones that should receive the call?

 

[JohanThiart]

Thanks for the Reply, Firewall Checker passes on server for all ports and on the remote network with devices/3CX App phones there is no outgoing restrictions to the Internet or to the 3CX Server. Should anything else be configured for Incoming Connections on firewall or NAT to the network where the 3CX Apps / Phones are used on since I cannot find any incoming ports to open on the link provided.

 

[JohanThiart]

I turned on Verbose Logging but there is no logging on Push related actions. I tried a missed call to a ring group to check if missed call notification will be pushed to phones using 3cx app and simulate the problem but there are no similar push messages logged on 3cx server. Is there a way to simulate a push message e.g missed call / dropped call via network or similar to see if server sends them via Google FCM to destination ring group or 3cx app on a phone ? If I search for the word "push" on the logs nothing is found.

 

[ChrisC_3CX]

If you have enabled Verbose logging, made a call to an android device, and that call actually worked, you should most definitely see the lines I mentioned earlier in the system service log. Did these calls fail perhaps? If yes, let me know so that we can take a closer look.

 

[JohanThiart]

I Enabled Verbose logging, sent some messages on 3CX, did outbound calls, inbound calls and missed calls to 2 different ring groups. I could not simulate the problem on the specific ring group since it occurs randomly at certain times during the day. The verbose log has no PUSH messages in it. Which Log file can I send to you to check why push messages are not done ? If I click on Logs there are three folders Global / Instance / Linux Syslog

If I download the FCM Toolbox on Android app and test it on devices for ring group where random calls do not ring during the day it shows Online and connected to Google FCM servers as a test.

 

[ChrisC_3CX]

As mentioned you need to check the 3cxSystemService.log

3. From the "Management Console >> Dashboard >> Activity Log" section go to "Logs >> Instance" and download the 3cxSystemService.log.

If you still cant find anything, send me the 3CXManagementConsole.log and the 3cxSystemService.log via PM.

 

[JohanThiart]

Thanks for the reply, I have sent the log files as a PM. The server also crashed (probably due to the verbose logging) after 8 hours of logging and I had to reboot the AWS Instance this morning to get it up and running again.

 

[ChrisC_3CX]

I've had a chance to look at the logs and replied back via PM with what I've found.

Verbose should not cause the system to crash unless of course you run out of Disk space so I recommend you have a quick look. Do turn Verbose off for now.