Solved Intermittent 2-10 second audio dropouts.

[scan-plastlv]

Hello, I would like to ask, if anyone has ever dealt with something similar or might have some ideas that would point me in the right direction.

Since around the start of this year, I'm dealing with complaints about intermittent 2–10 second audio dropouts during VoIP calls on 3CX v20(latest stable), where outbound RTP from 3CX to the SIP trunk abruptly carries encoded silence (G.711 PCMA, represented by repeated d5 bytes).

Observations:

• Packet Capture (done on the PBX):
  • Prior to the dropout, RTP packets contain normal audio data.
  • At dropout onset, packets from 3CX to the VoIP provider switch to a payload of d5d5d5…, which is the codec’s representation of silence. The phone->PBX RTP stream looks fine.
  • RTP sequence numbers remain consecutive, but once the audio resumes, they start over.
  • In longer dropouts, the phone→PBX stream continues for a few seconds, then also halts until both directions resume.
  • Around the onset of sound loss, no other network traffic that would be suspicious is present(such as SIP signalling), just the RTP streams.
  • When the audio returns, it seems to be offset/delayed slightly. It arrives in the correct phase from the VOIP provider, but the 3cx->phone stream has it's phase shifted. Once it unfreezes, it empties out a half-second of previously stuck audio(it's buffer?) which might be related to the offset.
• System & Network:
  • Network conditions, NAT, and firewall settings have not been changed for some time, and are pretty simple and straightforward.
  • 3CX logs (even in verbose mode) show no SIP re-INVITE, UPDATE, or other shenanigans that could disrupt things around the time of the drop-out.
  • Monitoring inside the VM (CPU, disk, I/O, and scheduling latency) shows no evidence of resource starvation or hypervisor-induced freezes.
  • PBX has an adequate 4gb of ram that sits half empty, and 4 cores of an AMD EPYC 7313P assigned to it.

Things I've tried:

• Enabling "PBX delivers audio" - no difference
• Looking at verbose logs at the moment of the drop-out, nothing seems to stand out.
• Ruled out the provider side of things, since it's the PBX that start's sending out RTP packets that are just silence.
• Using a soft-phone - no difference.
• DSCP and Bandwith QoS settings to prioritize outgoing packets from the PBX. Just improved jitter and other QoS metrics slightly.
• Moved 3cx to a different host, that is not busy. No perceptible changes.

Here's an image of it happening, when looking at the packet capture.
Bright green marker: onset of the dropout
Dark green: PBX -> Phone
Brown: PBX -> VOIP Provider
Dark-grey: VOIP Provider -> PBX
Blue-gray: Phone ->PBX

I'm getting a strong feeling that this might have something to do with the 3cx media handler. But I'm not sure how else to debug it besides the regular logging mechanism.

Thanks for reading and I'll be grateful for your insights!

 

[scan-plastlv]

After further experimentation, I have deduced that the problematic factor in my setup was CEPH. After changing the virtualization storage to a local SSD, instead of CEPH, I have not had any reports of this problem for well over a month.

Overall, this is surprising, considering that Netdata graphs did not indicate any substantial I/O pressure.

 

[fortech]

Footnote- thank you for posting your findings (!) This seems maybe to be an illustaration of - how voip is quite sensitive to latency, when compared to some other things/services you may run on the same virtualization environment (possibly?) so hopefully this can be instructive/resource for others in future about - importance of storage used for VM running 3cx (?!)

(ie, ceph as a storage pool type presumably has more complexity / more dependencies than vanilla SSD backed storage pool?)

and possibly not sure if you can tag your thread now as 'solved' so it shows up in forum thus / and others know it is a source of a solution to refer to in future (!?)

Thank you!

Tim