Solved Plivo - Stir/Shaken issues - some calls come through, others do not - Plivo support stumped

Status
Not open for further replies.

jpercival

Forum User
Joined
Aug 6, 2019
Messages
68
Reaction score
12
For cost purposes we use 2 different SIP providers (Flowroute and Plivo) to handle calls to our direct lines. 80% of our incoming calls are routed through Plivo and 20% are routed through Flowroute. A couple weeks ago we noticed that a significant number of incoming calls on our Plivo DIDs stated to pass through bad/extra caller ID info (ex. verstat=TN-Validation-Passed is added to the name or phone number, making it impossible to call back using redial, etc.), but not all of them. We were able to narrow it down to callers with T-Mobile (not Sprint surprisingly, also not Google Fi, which also uses T-Mobile towers).

We opened up a ticket with Plivo with EXTENSIVE logs of unaffected calls and affected calls, hoping it would be an easy fix on their end. After 9 days this problem still wasn't fixed. When I randomly looked at our incoming call log it was clear to see that we no longer had any calls with "verstat=TN-Validation-Passed" appended to them. I assumed the problem was fixed and attempted to make a test call to my DID using a T-Mobile phone to verify.

This is where the huge problem starts. The call did nothing. It stayed on the "Calling" status on the cell phone screen. No time counter or anything. After accidentally setting the phone down in this state the call went through after TWO FULL MINUTES. This has been verified on numerous different cell phones with the same behavior. Verizon phones don't have this issue. Sprint phones don't have this issue. My crappy Google Fi phone doesn't have this issue.

-recap-
I can call my desk phone (Plivo) from my cell phone and the call goes through perfectly.
I then call the same phone from my coworker's T-Mobile cell phone and it sits on the "Calling" screen on his iPhone for 2 minutes, connects the call, starts counting call time, and then my phone rings.
My coworker's same T-Mobile phone calls one of our DIDs handled by Flowroute and the calls goes through just fine.


After discovering the fact that we were clearly unknowingly missing out on a number of incoming phone calls we created a new critical ticket with Plivo. They have been spinning their wheels for over a day and are now starting to throw in language indicating that it's our 3CX server's fault for not responding. They indicate that it's likely related to Stir/Shaken but are still looking into it while we continue to miss calls blindly.

I did see on the 3CX forum that this "verstat=TN-Validation-Passed" issue has to do with Stir/Shaken, but don't know enough about that to even begin to think I could troubleshoot anything on our end.


Anyone have any ideas on this? Simply waiting for Plivo to figure it out seems like the only thing to do given the call behavior, but doing nothing is not a good look to our bosses.
 
Hi @jpercival

It is true that the "verstat=TN-Validation" messages have to do with STIR/SHAKEN but 3CX has not implemented anything regarding this yet. We basically don't have a mechanism to read the attestation level of STIR/SHAKEN yet but that should not stop the calls from coming in and as you describe when the calls did have the "verstat=TN-Validation-Passed" attached to them they came in fine but the issue was with the caller ID.

Since Plivo stopped attaching the "verstat=TN-Validation-Passed" info to the call you have been having this issue which tells me they did something on their end which also caused your issue with the delayed calls. I am not saying this is Plivo's fault as we do not have enough data but it is something that needs looking into.

So what you can do is set the PBX to verbose logging level and also run a packet capture while you are replicating the issue. Navigate to your dashboard --> Activity Log --> Settings and set logging level to Verbose. Then start a capture from the same screen and make a call from T-Mobile. Wait for the call to come in.
Stop the capture and save it and then open it to find the call. Check when the call came in and when it rang your phone. The activity log will show the same thing. This will tell you if the call is delayed from Plivo or if the call arrives normally and it is delayed by 3CX.

Also if you have a packet capture showing the "verstat=TN-Validation-Passed" in a call coming from Plivo send me a p.m as I would like to see how they implemented it.
 
Hey Yiannish,

I'd like to verify what jpercival is saying. We also use Plivio and any call that is being marked as "Verified" or "Not Verified" will result in 3cx NOT replying to the SIP invite. Calls that comes through as "Not Applicable" will be passed along just fine and the call will be normal.

There are two differences:

One is the X-Plivo-Stir-Verification field/value. This can either be Not Applicable, Verified, or Not Verified.
For calls that are verified, they are attaching "callerid" <sip:+11112223333;verstat=TN-Validation-Passed@*.*.*.*:5060> to the P-Asserted-Identity field.
 
  • Like
Reactions: jpercival
Hi @JSlow

I would like to see a capture of an incoming call from Plivo to see what is going on. Would you be willing to provide us with one?
 
  • Like
Reactions: jpercival and JSlow
Yes. It's a pretty busy day and I just got another support message from Plivo stating that their engineers are still looking into it.

They also said that they just had another support ticket come in from a 3CX user saying that they are no longer able to get calls from T-Mobile users.

I'll try to get logs when I can but I generally prefer to do log capture and testing after our business closes since the logs can get really difficult to parse through during business hours (and I'm getting my butt kicked right now).
 
Yeah I opened a ticket with Plivio a few hours ago too, jpercival. Haven't heard back but I basically stated it's only calls that are marked as "Verified" or "Not Verified". The funny thing is, STIR calls that were "verified" a few days ago were being passed through correctly/3cx was responding. I would much rather the caller ID be messed up than flat out not receive any T-mobile calls.
 
Yeah I opened a ticket with Plivio a few hours ago too, jpercival. Haven't heard back but I basically stated it's only calls that are marked as "Verified" or "Not Verified". The funny thing is, STIR calls that were "verified" a few days ago were being passed through correctly/3cx was responding. I would much rather the caller ID be messed up than flat out not receive any T-mobile calls.
Are you guys able to receive T-Mobile calls right now?
 
Are you guys able to receive T-Mobile calls right now?
Nope (I also have no way to test because no one in our company uses T-Mobile). However I will say it isn't exclusive to T-Mobile. Though, the majority of failed calls are a T-Mobile number. I've seen about 2 or so Verizon numbers also fail when their STIR status was verified. Same goes with verified Comcast numbers.
 
Last edited:
Decided to do the capture right now.

File is over 40 MB and the capture was started a few seconds before the call was started and stopped a few seconds after the call was answered and disconnected. Will send a direct OneDrive link to you guys.
 
Just do not put it publicly in the forum. Send it to me in a p.m. Btw I also contacted Plivo regarding this and I am waiting for a reply. I will let you know once I hear something
 
  • Like
Reactions: jpercival
Hi @JSlow

I would like to see a capture of an incoming call from Plivo to see what is going on. Would you be willing to provide us with one?

Had to send you a direct OneDrive share link since the capture was too large to upload. Link sent. Let me know if you have any questions.
 
2021-06-24 11_15_11-Wireshark · SIP Flows · dump.pcap.png


Here's some of the info in Wireshark pertaining to one of the test calls if anyone else is interested. It is indeed clear that the 3CX server is made aware of the call being placed within seconds of hitting SEND. The recipient (me) doesn't get the call in this example for almost 1.5 minutes.
 
Update from 3CX. Sounds like they are trying to get ahold of someone at 3CX directly and have been unsuccessful thus far. Here's the most recent message received from Nick, who has been very helpful:





Hello Jeff,

Thank you for providing the PCAP from your call. Normally when a call comes through from any SIP PCAP this is what we would normally see for a successful call

Plivo ---> SIP Invite (Header Key Value Pairs have specific details regarding the call, Body Key Value Pairs help us determine the CODEC that will be used for the audio stream) is sent to 3CX
3CX <--- address sends back a 100 trying message, a Ringing status, or a 200 Status OK to Plivo
Plivo ---> Sends an acknowledgement that it received the 200 Status OK to 3CX

Past this point the call is connected and audio is established.

What is happening currently is the following:

Plivo ---> SIP Invite to 3CX
Plivo ---> SIP Invite to 3CX
Plivo ---> SIP Invite to 3CX
Plivo ---> SIP Invite to 3CX
Plivo ---> SIP Invite to 3CX

Plivo <--- 408 Request timeout is sent back to the Carrier

When Plivo is sending the Invite we are not receiving a response from 3CX. This is causing the request to time out and in some cases you will not get a connection and the call will fail. In other situations, after multiple attempts to connect, Plivo receives a response back from 3CX and the call is connected, this is situations where you witness a long delay before the call connects.

Now the problem here as mentioned before, is not necessarily on 3CX not sending a response, it is very likely the issue is with the data being sent in the header fo the SIP Invite. What our engineering team has been working on has been to identify what in the header might be causing 3CX to not respond.

Our engineering team had been trying to reach out to 3CX, however in answer to a forum post I see that was started about this issue today 3CX reached out to work with us on this issue. We have been trying to get in contact with them since they reached out but have not been able to connect yet. We are still working on the issue internally to understand what is occurring so that we can work on implementing a fix for it as soon as possible.

Please let me know if you have any questions about the above.

Regards,
Nicholas
 
  • Like
Reactions: YiannisH_3CX
After some quick initial testing it looks like Plivo has resolved the issue!
 
Setting this one as resolved since the issue was resolved from Plivos side. Please do not hesitate to create a new thread if you face any more issues.
 
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
112,324
Messages
591,897
Members
165,476
Latest member
awan05