Video conference quality issues in Germany

[weimerd]

Hi,

we recently see a downgraded conference quality in a specific time slot of 8:30 to 10:30 a.m. in Germany region.
Before and after that time slot video conference quality is good. But within that time slot we get bad video and even audio quality.
It is a reccuring issue. We tracked that now for several days.
Is there anything we could do from our side?

We tracked that with three clients now. Seems a MCU issue with EU servers.

Any preview of on-premise MCU available?

Best regards

David

 

[weimerd]

same today. In the mentioned time window 8:30 to 10:30 a.m. Berlin time performance of webmeetings are reported to be bad. I tracked three customers now having the same problem. Seems like MCUs in Germany region show some performance issue.

Reported behaviour in webmeetings (only within this time slot): bad video and audio quality. Sometimes video freezes for a while. Randomly participants get disconnected. Screen share quality is unuseable.

I checked it in person during two of their meetings and can confirm the behaviour.

System is up to date. I even updated one of the instances to latest RC. no change.
Servers are running in azure with low CPU and RAM usage.
Users are connected from various locations. So it isn't a local internet connection issue as well.

Would love to get some feedback, even if it's "Yeah, we have some performance issues in this region actually at busy hours.".

Thanks in advance

 

[weimerd]

I investigated a bit further and tracked that especially in time window 9:45 to 10:15 a.m. Berlin time quality get's really worse. Video stutters for all participants. Report was from two clients as I made them to detail the issue a bit more.
I joined one call for technical purpose and can confirm: video stuttered for all participants.

Of course Europe as MCU region has been chosen.

Hope this helps to work on the issue. These two clients are starting their service meetings with their clients at this time and are not very happy at the moment.

 

[LeonidasG_3CX]

Hi,

When users first experience this issue, could you ask them to tell you their CPU usage on their machines?

The MCUs you're using in Europe are dedicated machines in both Amsterdam and Frankfurt.We've seen no issues whilst monitoring them and there's no scheduled maintenance by the provider or us during this period. We also make sure that provider maintenance doesn't affect users when it does occur.

We're in fact working on tools that will help you understand who is affected by what during a meeting in order to help understand why you're experiencing low video quality. Hopefully once these improvements are out, they will help in understand why your users are experiencing what they're experiencing.

Additionally, our Video Conferencing solution currently knows when someone in the meeting is unable to send or receive video and automatically lowers their reception and transmission quality in order to prevent constant image freezing. The same can occur for 1 or more users in a meeting depending on their computer's ability to send/receive network data & to process multiple videos at the same time. CPU on every user's computer is extremely important and needs to be checked during image freezes.
If your CPU is peaking during these moments, it's the cause of your low quality. Similarly, someone uploading large files over the internet in the same office and no QOS in place in your network can cause the same problems.

I would suggest first checking these. We're seeing on our side that everything looks ok so far. We haven't had any other reports of this which makes me think that one of the above mentioned items might be causing problems for your users.

 

[Ronin]

We're in fact working on tools that will help you understand who is affected by what during a meeting in order to help understand why you're experiencing low video quality. Hopefully once these improvements are out, they will help in understand why your users are experiencing what they're experiencing.

Hello,

This is very interesting. Is there an ETA for the web conference tools?

 

[LeonidasG_3CX]

Hello,

This is very interesting. Is there an ETA for the web conference tools?

The only timeline i can give you is month/s. We are currently working on them.

 

[weimerd]

Hi,

When users first experience this issue, could you ask them to tell you their CPU usage on their machines?

The MCUs you're using in Europe are dedicated machines in both Amsterdam and Frankfurt.We've seen no issues whilst monitoring them and there's no scheduled maintenance by the provider or us during this period. We also make sure that provider maintenance doesn't affect users when it does occur.

We're in fact working on tools that will help you understand who is affected by what during a meeting in order to help understand why you're experiencing low video quality. Hopefully once these improvements are out, they will help in understand why your users are experiencing what they're experiencing.

Additionally, our Video Conferencing solution currently knows when someone in the meeting is unable to send or receive video and automatically lowers their reception and transmission quality in order to prevent constant image freezing. The same can occur for 1 or more users in a meeting depending on their computer's ability to send/receive network data & to process multiple videos at the same time. CPU on every user's computer is extremely important and needs to be checked during image freezes.
If your CPU is peaking during these moments, it's the cause of your low quality. Similarly, someone uploading large files over the internet in the same office and no QOS in place in your network can cause the same problems.

I would suggest first checking these. We're seeing on our side that everything looks ok so far. We haven't had any other reports of this which makes me think that one of the above mentioned items might be causing problems for your users.

Dear Leonidas, thanks for coming back to me. Tools would be great to better understand in future.

It's not the CPU or hardware issue. All users get the same message (bandwith warning with the hint to maybe stop video and go on with audio only).
The used hardware is various. some are even using way overpowered machines like Mac M1. I checked with mine, no CPU peak when it comes to problems.
Weird that it just happens within this time slot. We could all do a video call now without any issues. It's just within this specific time slot.

Freeze of video happens for all participants. They all do have different hardware and different internet conneciton speeds.

I don't know how the load balance is working at MCU side. I guess that within that time slot many users are starting video calls imn Germany, generating loads of traffic to the MCUs. From our side it looks like MCUs are running against a limit.

 

[LeonidasG_3CX]

Dear Leonidas, thanks for coming back to me. Tools would be great to better understand in future.

It's not the CPU or hardware issue. All users get the same message (bandwith warning with the hint to maybe stop video and go on with audio only).
The used hardware is various. some are even using way overpowered machines like Mac M1. I checked with mine, no CPU peak when it comes to problems.
Weird that it just happens within this time slot. We could all do a video call now without any issues. It's just within this specific time slot.

Freeze of video happens for all participants. They all do have different hardware and different internet conneciton speeds.

I don't know how the load balance is working at MCU side. I guess that within that time slot many users are starting video calls imn Germany, generating loads of traffic to the MCUs. From our side it looks like MCUs are running against a limit.

Hi,

We'll run some tests within the timeframe you mentioned. Additionally, i can assure you that we keep a ~50% max resource buffer for our MCU servers and that they are absolutely always kept very far from 100% usage to avoid exactly what you're mentioning.

You mentioned that the video freezes. Does it actually fully freeze or does it continue with low quality or both? Also if it does freeze, how long do freezes generally last for? When you say "Freezes", can you describe these freezes for us? Does another person's video just go black, empty or is their image frozen or just low quality?

Lastly when you connect to an MCU/Meeting Server, if you check the Settings > Stats Page > Bottom of the page, what server does it say that you're connected to usually?

 

[weimerd]

Hi,

We'll run some tests within the timeframe you mentioned. Additionally, i can assure you that we keep a ~50% max resource buffer for our MCU servers and that they are absolutely always kept very far from 100% usage to avoid exactly what you're mentioning.

You mentioned that the video freezes. Does it actually fully freeze or does it continue with low quality or both? Also if it does freeze, how long do freezes generally last for? When you say "Freezes", can you describe these freezes for us? Does another person's video just go black, empty or is their image frozen or just low quality?

Lastly when you connect to an MCU/Meeting Server, if you check the Settings > Stats Page > Bottom of the page, what server does it say that you're connected to usually?

Lastly, can you explain

Hi,
"Freeze" in that context means video freezes, then after 10-15 seconds switchs to very low quality, freezes again, changes quality again. That happens all the time.
Video and audio does not sync anymore, big delay in video to audio.
So video is not useable at all. Audio quality keeps more stable and runs constantly more or less. Very rarely there is some audio cut for some seconds.

Happens for all participants. Internet connection is good at every point. Everyone has a bandwith of minimum 50 down, 20 up.
Most participants even run higher bandwiths. Two even 1Gbit up and down.
Hardware varies, but all machines are more than capable enough, no CPU limit hit. I checked it with three end user machines, they all run at 10-20% CPU.

I check statistics for tomorrow regarding the used MCU server. Actually it says mcu-wmr-eu-07208.3cx.net, but no issues at the moment. Anyway I got report for today that there where issues again for at least one team.

Tommorow I come back with the actual MCU name when it comes to issues. Will let them report.

 

[weimerd]

Hi,

We'll run some tests within the timeframe you mentioned. Additionally, i can assure you that we keep a ~50% max resource buffer for our MCU servers and that they are absolutely always kept very far from 100% usage to avoid exactly what you're mentioning.

You mentioned that the video freezes. Does it actually fully freeze or does it continue with low quality or both? Also if it does freeze, how long do freezes generally last for? When you say "Freezes", can you describe these freezes for us? Does another person's video just go black, empty or is their image frozen or just low quality?

Lastly when you connect to an MCU/Meeting Server, if you check the Settings > Stats Page > Bottom of the page, what server does it say that you're connected to usually?

During problematic time slot MCU is mcu-wmr-eu-07360.3cx.net (Netherland) for today. So for some reason it won't take a server in Germany.

 

[LeonidasG_3CX]

I've gone ahead and made a few changes to see if somehow the AMS region is part of the problem. AMS has been replaced with FRA e.g Frankfurt.

Let me know if the same quality issues occur using Frankfurt servers.

 

[weimerd]

I've gone ahead and made a few changes to see if somehow the AMS region is part of the problem. AMS has been replaced with FRA e.g Frankfurt.

Let me know if the same quality issues occur using Frankfurt servers.

Today I got no issue report. Joined one meeting for test purpose and could not find any problems at all. Server was shown with location "Germany" this time.

 

[LeonidasG_3CX]

Let's do this a few more times and see what happens to verify this completely. Also i'm not sure i asked in previous comments, but has your IT department setup QOS for Video Conferencing? I think i mentioned this in a mail i contacted you with.

If your IT department adds this address as a target for QOS v18-vc-qos.3cx.net then people in your office who use Video Conferencing will be guaranteed priority in media delivery over let's say someone who is uploading large files to a Google Drive or is doing a download or upload intensive process at the same time that you're in a video conference.

Also it could be worth asking your IT engineers if they have QOS for something else that could be causing lower media delivery priority for Video Conferencing. In any case it's worth checking this part.

 

[LeonidasG_3CX]

Today I got no issue report. Joined one meeting for test purpose and could not find any problems at all. Server was shown with location "Germany" this time.

Has everything been fine so far?

 

[weimerd]

Has everything been fine so far?

Still some webmeetings will use MCU in Netherland. Anyway actually I did not see any performance issues. Will investigate further including upcoming week and will come back with a conclusion then.

 

[weimerd]

Has everything been fine so far?

After some more investigation I could sum up:

If MCU with server location in Germany (Frankfurt) is been used then no issues.

MCU with server location AMS Netherlands has performance issues up to audio cuts. I made a team stopping a meeting that what almost unusable for every participant. MCU was AMS in Netherlands. Quality of video was not acceptable at all and freezes constantly for all members. Audio stuttered and was not usable.
It was a team of 6 persons speaking, including me as technical supervisor.

After restarting MCU in Germany was chosen. Audio and Video quality was perfect.

I don't know why MCU in Netherlands are chosen as all participants are located in Germany. But that's the root of the issue I guess.

Please if it is possible make sure to route German participants to MCUs within Germany only. That would highly rise the acceptance of 3cx WebMeeting. One customer switched to MS Teams already until there is a solution.

Thanks Leonidas for your investigation so far. Any additional information needed just let me know.

Best regards

David

 

[LeonidasG_3CX]

After some more investigation I could sum up:

If MCU with server location in Germany (Frankfurt) is been used then no issues.

MCU with server location AMS Netherlands has performance issues up to audio cuts. I made a team stopping a meeting that what almost unusable for every participant. MCU was AMS in Netherlands. Quality of video was not acceptable at all and freezes constantly for all members. Audio stuttered and was not usable.
It was a team of 6 persons speaking, including me as technical supervisor.

After restarting MCU in Germany was chosen. Audio and Video quality was perfect.

I don't know why MCU in Netherlands are chosen as all participants are located in Germany. But that's the root of the issue I guess.

Please if it is possible make sure to route German participants to MCUs within Germany only. That would highly rise the acceptance of 3cx WebMeeting. One customer switched to MS Teams already until there is a solution.

Thanks Leonidas for your investigation so far. Any additional information needed just let me know.

Best regards

David

Since a bit over a week now, we've had only Frankfurt MCUs to be used for Europe. There have been no Amsterdam/NL based MCUs during the period where we've been receiving your feedback. If during this period everything was fine, we'll consider the issue being a routing issue with AMS/NL and will permanently avoid those.

Have you been assigned any AMS/NL MCUs this week and the last? It shouldn't be possible as there have been none. If you can confirm this, we can consider this case closed. Otherwise we'll look further into it as there are a few other solutions we can try for you.

 

[weimerd]

Since a bit over a week now, we've had only Frankfurt MCUs to be used for Europe. There have been no Amsterdam/NL based MCUs during the period where we've been receiving your feedback. If during this period everything was fine, we'll consider the issue being a routing issue with AMS/NL and will permanently avoid those.

Have you been assigned any AMS/NL MCUs this week and the last? It shouldn't be possible as there have been none. If you can confirm this, we can consider this case closed. Otherwise we'll look further into it as there are a few other solutions we can try for you.

yes, we got AMS/ NL several times this week. Today at 10 a.m. Berlin time we got the last connect to AMS/NL. Seems some routing issue happens from your side