Incoming calls with silent voicemails & no sound

[epowers]

We have had quite a few reports of incoming calls from external callers where there is no sound when receiving the call (and/or) no sound in voicemails (voicemails are ~20 seconds long). I've taken a look at our Twilio SIP logs, and I'm seeing cases where there are "request timeouts" when our SIP provider is attempting to reach our on-premise 3CX system. This issue is intermittent, and most calls work properly, but some do not.

I have PCAPs from Twilio (SIP provider) for these instances, but I'm not seeing anything in them that is abnormal. I've also recently turned on verbose logging in 3CX to try to capture issues that I find there, but no issues have happened since I've enabled the verbose logging. When I get that information, I will post it here. But here's the log for the specific calls that the user reported yesterday:

06/29/2022 11:17:58 AM - [CM503003]: Call(C:206): Call to <sip:[email protected]:0> has failed; Cause: 487 Request Terminated/INVITE from 127.0.0.1:5063

06/29/2022 11:17:57 AM - [CM503003]: Call(C:206): Call to <sip:[email protected]:0> has failed; Cause: 487 Request Cancelled/INVITE from OUR.LOCAL.IP.ADDRESS:5060

06/29/2022 11:17:30 AM - [CM503003]: Call(C:205): Call to <sip:[email protected]:0> has failed; Cause: 487 Request Terminated/INVITE from 127.0.0.1:5063

06/29/2022 11:17:30 AM - [CM503003]: Call(C:205): Call to <sip:[email protected]:0> has failed; Cause: 487 Request Cancelled/INVITE from OUR.LOCAL.IP.ADDRESS:5060

Edit: Just had an incident happen, and I've attached an obfuscated log.

I spoke with the caller who was calling in and experienced the issues, and they are using a Verizon cell phone. The caller mentioned that they got a "Provider not available" message when they attempted to call.

We are using the latest release (18.0.3.461), and have all proper ports open, and 90% of calls are functioning properly... Just 10% or so are not connecting correctly.

• 3CX Version, - 18.0.3.461
• Server OS, 3CX OVA Image for VMware.
• Is the 3CX Server Hosted and where? - On Prem
• IP Phone Make/Model/Firmware version, Yealink T54W - 96.86.0.72 & DesktopApp 18.8.508
• Provisioning Method: Local
• Trunk Provider or VoIP Gateway Make/Model, Twilio Elastic SIP Trunk
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

Thanks in advance for helping me figure this tricky issue out!

 

[cobaltit]

When you say the firewall checker passed, does that mean recently since this incident started? And are you doing any restricting at all for inbound traffic?

 

[epowers]

Our firewall is configured such that we block all ports other than those that are coming from Twilio (our SIP provider) based on their list of IPs (all IPs listed here: https://www.twilio.com/docs/sip-trunking/ip-addresses). If I turn off this rule, then the Firewall Check passes without issue, but then I get hammered with attacks from the outside:





Thanks for the reply and looking forward to your thoughts.

Thanks again!

 

[SethTalley]

We were having shenanigans several months ago where occasionally our customers would call and both sides would hear a fax machine hunt tone over their conversation. I did a reverse number lookup and determined that every call was from a cell number serviced by Verizon Wireless. I can't recall what hunting I did at the time, but I determined that Verizon had recently done an overhaul to bring their terrestrial and wireless protocols closer in line, and hypothesized that the addition of T.38 signaling was causing our carrier (Voip.ms) to see certain Verizon wireless numbers as fax machines (note that voice data was also passed).

We probably dealt with a half dozen of these issues over three months, enough to be annoying but not enough to do anything about. Eventually the problem went away.

This morning one of our patients made a 3 minute call to our paging system, which takes a voicemail if nobody picks up. Time-to-prompt is 42 seconds outside of office hours (they ring directly to the phone tree) or 1 minute 1 second inside office hours. The patient was leaving a 2-minute voicemail after the prompt, but it never hit the voicemail extension, never made a recording, and never emailed that recording to a monitored inbox.

I saw this while investigating:

https://www.3cx.com/community/threads/voicemail-not-saving-randomly.114498/

As well as this thread. The phone number we're having difficulty with is Verizon, the phone number in that thread is Verizon, this thread is Verizon.

What's the likelihood that Verizon is sloppy with their protocols and confusing 3CX? So far this has happened only once and there's no good way for me to do further testing. We'd probably have to leave the system in verbose mode for a week or two (it's 3CX-hosted) to get anything useful.

 

[cobaltit]

Hey you tried removing the blocking and seeing if the issue persists?

 

[SethTalley]

Disregard. Our immediate problem, at least, was a patient who thought "repeat" meant "verify". She pressed pound after leaving a voicemail, then got confused and opted to "repeat" it, then got more confused and hung up, thereby deleting her voicemail.