Solved Random Caller Silence - Update to the case

[siscom]

Good Afternoon Colleagues
I hope you are doing good and safe . I have opened before a case for this issue and below its link:

https://www.3cx.com/community/threads/random-caller-silence.82515/#post-383172

Thanks to @ChrisC_3CX whom helped me till we close the previous case.

There is new variable that lead me to open this new case.

I have make deep investigation by capture some pcap files using capture tool at gateway.

The first capture file (Capture 1) i have attached in the previous thread.It has 2 VOIP Calls . At the time of the capture you should hear 3CX system prompt message which always played when there is no agent available . Kindly find the below notes:

• The first call has the issue where the caller can not hear anything. When open wireshark to hear call from wireshark , I noticed that there is a noise in direction from VOIP gateway to 3CX.
• The second call is perfect where the caller can hear 3CX system prompt message.
• The IP Address of 3CX is 192.168.1.250 while the ip address of gateway is 192.168.1.251.
• I can hear the calls by open wireshark==> Telephony ==> VOIP Call.

The second file (Capture 2 & Capture 3 ) are normal calls with no noise.

The Fourth file (Capture 4) contains one VOIP call .The call has the issue where the caller can not hear anything. When open wireshark to hear call from wireshark ,you will notice that there is a noise in direction from VOIP gateway to 3CX.

What i noticed after deep investigation the below:

• For the calls that has noise , In the direction from gateway to 3CX , you can find "out of sequence" packets & "Wong Timestamp" packets with high percentage which is not found at direction from 3CX to gateway.
• For the calls that has no noise , In the direction from gateway to 3CX , you can find at some calls "out of sequence" packets & "Wong Timestamp" packets with low percentage and some calls no errors at all . The direction from 3CX to gateway still perfect with no errors.

So may i know if those "out of sequence" packets & "Wong Timestamp" packets has any relation with noise found in direction from VOIP gateway to 3CX ?

Best Regards

 

[NickD_3CX]

The short answer to your questions is 'yes'. out of sequence means generally that Wireshark detects that some packets are missing.
Each voice packet is numbered 1...2...3...4...etc...
If Wireshark see packet 1, then packet 3, it will flag 3 as "out of sequence" because it never saw a packet 2.

Now in your case, it looks like every 2 packets, the 1 is lost, Wireshark detects ~50% packet loss for the audio.

 

[siscom]

Hi Nick
Thx for your replay . So does the VOIP gateway is the device responsible of this out of sequence or maybe other factors related to mobile operator , landline operator or other factors ?

Best Regards

 

[NickD_3CX]

I would first contact the manufacturer of the VoIP Gateway and take it from there.

 

[siscom]

Hi Nick
I have make more captures using wireshare and i would like to share with you the attached captures.

Those captures are for successful calls where all calls has no noise and you can hear the calls clearly with no noise inspite of the fact that there is "out of sequence" packets & "Wong Timestamp" packets with the same high percentage (around 50%) found at calls with noise.

Do you have any explanation for this ?

Best Regards

 

[NickD_3CX]

Hi @siscom ,

I don't have much of an explanation really to be honest, not, but for sure 'out of sequence' is not normal and not a good thing.

I think though you are over-thinking this.
The facts are simple:

• When you listen to the RTP stream is Wireshark, the audio stream is distorted on some calls, but on some other it's fine.
• The distortion is coming from the GW IP

With the above, you need to take this up with the GW Support Team.
Have you contacted them? What have they said?

 

[NickD_3CX]

Upon a closer review of the capture, it seems that on the calls that have the problem, the PBX is receiving duplicate RTP packets which could be causing this.
Now the way I see it, either the GW is sending the duplicate packets, or something on the network (switch?) is duplicating the packets.

One more thing you could try is swapping out your switch and all in-between networking devices to see if the issue is being caused there.

 

[siscom]

Good Afternoon Nick
Thx for your investigation . I didn't contact the GW vendor yet . I will contact them with the results we have achieved so far and will let you know their feedback.

Thx for your support.

Best Regards

 

[siscom]

Good Afternoon Nick
I have some updates regarding this case:

• I have contacted the VOIP gateway vendor and he informed me that the problem may be compatibility issue with other party (ISP E1 device).
• I have done more investigation. I have connected ISP E1 device to CISCO router with E1 interface . The CISCO router has also FXO interface . This FXO interface connected to FXS gateway which has analog phone connected to it
• The call flow as follow:
  • Call received by ISP E1 device forward to CISCO E1 where CISCO is configured to forward all traffic to FXO interface. FXS port connected to CISCO FXO port is configured to automatically ring the analog phone once call received.
• Using the above test , I can exclude at all the VOIP gateway and test the 31 E1/PRI Channels.
• Let me share with you the test results:
  • For 2 or 3 calls , There is about 5 to 6 seconds of silence then call disconnected.
  • For 2 or 3 calls , The call didn't established at all.
  • Other calls working as expected.
  • The above is randomly happened so at sometimes E1 channel 2 works fine and at the next test call received at E1 channel 2 may disconnect or didn't establish.

So based on the above , What is your opinion ?

Best Regard

 

[NickD_3CX]

Hi @siscom ,

So if I understand correctly, you have completely 'excluded' the PBX from the call flow to test if it is something with the GW.

Based on your results, even with this setup, you still have problems.
I insist, it could be something with your switches where some packets are getting duplicated and sometimes some are not even delivered.

By the time that, based on your setup, this does not seem to be related with the 3CX software, I think you need to start checking the logs of the devices one-by-one to see where the issue is being caused, but if I were you, I'd start by replacing the switch with something else, even if it's a dumb L2 switch to connect your VoIP GW with the rest of the network.

 

[siscom]

Hi Nick
Thx for your replay and your willing to help solve this issue. Let me share with you the below notes:

• In my test , I have excluded both 3CX & VOIP GW.
• In my test , NO IP network at all . No switches , NO IP Phones.
• In my test , The connection was: ISP E1 device ==> CISCO E1 ==>CISCO FXO ==> FXS gateway ==> Analog Phone. So in my test , I just convert E1 digital connection to analog connection to be connected to analog phone excluding any IP device.

Sorry if i didn't make it clear in my previous message.

Kindly let me know your thoughts.

Best Regards

 

[NickD_3CX]

Ah, this is interesting, sorry, didn't realize.

In this case I'm not sure I can advise you much further. While I do know and understand the inner workings of Gateways, I can't say I'm an expert on them.
At least for the scenario where calls aren't established at all, you could check the logs of all the in-between devices and start checking if the call arrived or not, so at least that way you can backtrack the issue to the device causing the fault.

 

[siscom]

Hello Nick
Thanks for your advice . I think this test has normalize one of the issues that i am facing which is the silence.

There is another issue where the call is hang or there is noise even in the 3CX recording.

Do you think that the above problem related to network/switch in spite of the fact that "out of sequence" packets & "Wong Timestamp" packets are in the direction from Gateway to 3CX only (What i mean that if there is a problem in network/switch , "out of sequence" packets & "Wong Timestamp" packets should be in both directions not only in direction from gateway to 3CX)

What is your opinion ?

Best Regards

 

[NickD_3CX]

I couldn't say for sure, but yes, it could be related.. This is one of the cases where I would suggest contacting our support team to have a close up look at the issue, but because you are not using a support VoIP GW, this is not an option.

 

[siscom]

Hi Nick
Thx for your great support in this case . Kindly fell free to close it.

I will open new case if there is major updates happened.

Best Regards