Performance issue with calls when huge read access on HDD

[ndyserinck]

Hello everyone,

For some time now, I have noticed that our phone calls are interrupted or jerky when our backup system executes a backup of files (over 600,000).
These files are stored on the same NAS (disk) as the 3CX VM, but in a separate partition.
Only this type of backup impacts our phone calls. Our backup software processes more than 4TB of VMs every night, and without a hitch, the problem only arises when it processes these 600,000 files.

Depending on the processing time, this can last up to several minutes. It's impossible for us to make phone calls properly in that moment.

Our VM configuration :
- Multiple VM Host (Vmware) hosting multiple VMs
- Storage on NAS with partition for VMs and for File Data (LUNs - iSCSI)
- Average disk usage = < 10% // Latency = < 2ms
- Peak disk usage (when backup file) = ~90% // Latency = ~100ms

- 3CX Enterprise V18.0 (Build 917)
- Call center use 24h/24 7d
- QoS enabled on Switchs
- Users options : PBX Deliver audio // Recording all calls (in VM directly - Archived later)

Has anyone ever encountered this kind of problem? How can intensive use (file reading rate) of the hard disk affect phone calls?

Thanks in advance for your help.

Kind regards,

 

[Charles_3CX]

To clarify, do you have a NAS that stores data on the same disk (different partition) shared with the 3CX VM?

And, you are wondering why, when a backup reads the disk to backup up 4TB of data, the 3cx system suffers issues?

 

[SteveITS]

I would guess "Recording all calls" is related here. Though you haven't mentioned CPU usage during this time.

I'd suggest having multiple partitions is actually a bit worse here, because the disk heads have to seek to two different file systems. If possible move the 3CX VM disk to a different set of disks/spindles.

 

[ndyserinck]

Hi Charles,

Yes the same disk (Raid 1) have 2 volumes (LUN) :
1) LUN for some VM (not all) but 3CX is on it
2) LUN for our Windows Share. Directly mounted in the Windows server VM

The issue occurs only when the backup works on the LUN which is connected to our Windows Server. It represent less than 500GB.
When the backup job works on all VM, there is no issue on 3CX. 3CX vm is also backupped and without any interruption.

@steve,

Yes i can move the VM but I am curious to know why it happen only with this kind of workload.
3CX CPU is never above 15%. NAS CPU is around 60% to 90% when backup works. The bandwidth used by it is around 250MBytes / second when it crawl all VM.

 

[ivank]

I would guess "Recording all calls" is related here. Though you haven't mentioned CPU usage during this time.

The recording process uses in-memory buffers and a background thread that writes them to the disk chunk by chunk. Lots of seeking might slow down this indeed but it should not interfere with the audio stream at all. So it should be something that tries to synchronously write to the disk and blocks in the middle of the audio processing where it normally shouldn't.

@ndyserinck,
What's your activity log setting? If anything above Low, try setting it to the minimum.

 

[ndyserinck]

Hi @ivank

Activity log is on debug, i forgot to switch it to a lower settings a while ago... i'll change it to Low.

This settings can interfere with audio stream when disk activity is very high ?

 

[ivank]

Of course it can as it spits out lots of text to the disk in all subsystems of the PBX and it is always a blocking write.

 

[ndyserinck]

Ok then...
Settings is now on Low. I will later restart 3CX because it's busy time now for our company.

Once it restarted, i will run the backup file job and monitor.
I will post the result here.

If it still mess with the audio stream, i will move the vm to another datastore.

 

[ivank]

This setting applies immediately, no need to restart anything. Please also don't forget to let us know how it went next night.

 

[SteveITS]

setting applies immediately, no need to restart anything

Side note, that's been my observation all along, but the dialog box says, "Logging level (restart required for changes to take effect)."

 

[SteveITS]

it spits out lots of text to the disk in all subsystems of the PBX and it is always a blocking write

Has 3CX considered a dashboard note or icon to indicate verbose logging is on?

 

[ivank]

Side note, that's been my observation all along, but the dialog box says, "Logging level (restart required for changes to take effect)."

Most likely an oversight that was left from much older versions of the PBX.

 

[ndyserinck]

Hi @ivank,
After a day with the logs settings to "Low" (without restarting) we did not have any issues.
I even did some extensive works and Full backup job with no problem. Disk usage was above 90% on the NAS and 3CX did not suffer from it.

I will check in the next few days to see if it happens again. If it does, I will move the VM to a different datastore.

Many thanks for your support.

Kind regards