Strange Hyper V Issue

[tworhythms]

I have been evaluating 3CX Version 9 for a couple of weeks now and absolutely love it!

This is my first 3CX install so please bear with me

My setup is as follows:
XEON E5520 with 8GB Ram
Server 2008R2 running Hyper-V on LSI Raid 1
2 TB on dedicated LSI controller in Raid 10 (4 Sata 7200rpm drives)
(2) Cisco SPA504G Phones as well as (2) 3CX Softphones

All of my virtual machines are installed on the Raid 10 drives.
My virtual machines are:
-Domain Controller 1 GB Ram (Server 2008R2) Dedicated 10/100/1000 NIC
-Exchange 2010 4 GB Ram (Server 2008R2) 10/100/1000 Dedicated NIC
-3CX 2GB Ram 10/100/1000 Dedicated NIC

When I try to run 3CX on a fresh 2008R2 Virtual machine the outgoing call quality, IVR and music on hold is fairly poor (i.e. clicking, popping and distorting).

This is where the scenario gets strange:
If I install 3CX on an existing Server 2003 Standard (2 GB 32-Bit version) Virtual using the same virtual switch as the 2008R2 machine and stored on the same Raid 10 setup the audio quality is great. No issues at all.

Also, if I install 3CX directly on the Hyper-V host machine (Server 2008R2) the audio quality is excellent as well.

So, this is what I've tried on the Virtual Server 2008R2 machine:
-Changing the properties of the Virtual and Physical NIC (i.e. IPv4 offloading, buffers, etc...)
-Changing to a different virtual switch
-Implemented QoS on the Server 2008R2 virtual
-Moved the virtual machine to a separate drive with a fixed VHD
-Changed audio codecs, even tried using K-Lite codecs
-Tried assigning 2 processors to the Server 2008R2 virtual instead of 1
-Shut down Exchange 2010 and increased memory allocation to the 3CX virtual (4 GB)
-Desktop experience and Quality Audio Video is installed
-Changed the heartbeat, time syncronization properties of integrated services
-Checked and double checked the firewall settings using the firewall checker
-Used Wireshark to capture and analyze SIP and RTP packets (nothing out of the normal showed up)

On a side note; if I install 3CX on an existing Windows 7 virtual machine using the same virtual switch (virtual NIC) the audio is terrible as well.

So, why is the audio quality so superb while running on the Server 2003 virtual machine as opposed to the Server 2008R2 and Windows 7 virtual machines, especially considering they all use the same virtual switch (not all at the same time though)? This would lead me to believe that the problem is not with networking.

Any help/thoughts would be truly appreciated. While I don't mind running 3CX on the Virtual Server 2003 machine, I would much rather have it running on a current OS (i.e.Server 2008R2).

Thanks everyone!

 

[abc123]

It appears that Win7 and Win 2k8R2 (both built using similar kernels etc.) are using a different virtual nic drivers.

We are running 3cx on a HyperV wish 1 processor and 1.5Gb Ram but Win 2k8 Enterprise 64 and a virtual nic. No problems at all. This also runs some other apps of ours such as team foundation server etc. Load is minimal. Quality and response is high.

Can you check your virtual nic? Do you have legacy adapter turned on in error? IF not, try turning it on to see if that makes any difference. Also double check your hyper v drivers (maybe a reinstall of the additions).

You can also look at cpu usage on the physical machine and the 3cx virtual machine when a call is made (Check ram usage too).

Do you have an anti virus on the 3cx machine? Is it trying to real time scan the audio?

 

[tworhythms]

Thanks for the suggestions.

Unfortunately, the issue persists. I am not running any anti-virus software on any of these virtuals until this issue is sorted out.

Today, I tried installing a fresh Windows 7 Ultimate x64 Virtual machine and assigned 2 GB to it. The exact same problem (random clicking in the outgoing audio, IVR and music on hold). Internal calls between extensions appear to work correctly, but calls through my Voip provider have the clicking (breaking up).

So, something doesn't make sense. Why can I make calls to the outside world with the Server 2003 virtual machine on the same host and with the same virtual switch without any issue? Ping times to my VoIP provider are very good (avg. 13ms) as I am close to one of their servers. 3CX Firewall passes all of the tests.

The only other thing that came to mind was that the Server 2003 machine used to be a physical machine and was converted to a virtual.

Is it necessary to have a physical sound card installed in the host machine? I wouldn't think this is the problem since I can install 3CX directly on the host machine without these audio problems.

During calls CPU usage remains fairly low (avg 9-13 percent) and memory stays at approx. 58 percent.

Has anybody else experienced this? I really do appreciate the help.

 

[robbie]

Whe have the same problems with server 2008.

 

[abc123]

As far as I am aware (and is the case in our setup with Hyper V) you DO need a physical sound card in the host machine.

 

[tworhythms]

Update -- abc123, I think you could be on to something.

I enabled RDP Audio passthrough on the problematic Server 2008R2 virtual machine and when I log into the machine via RDP you can see that the volume icon doesn't have the red "x" through it (i.e. RDP default audio device). Now, while I am in a RDP session and I make a call, initially the call is broken up and crackling but as soon as I start to move the volume slider on the Server 2008 virtual machine the audio is great and will remain that way for the duration of the call.

Going back to my Server 2003 virtual (which works flawlessly), the RDP audio device ALWAYS shows enabled and is set as the default sound device. This used to be a physical machine with a sound card. Perhaps the audio drivers remained while it was converted to a virtual machine? Either that or Server 2003 handles the sound drivers completely differently (which probably isn't a stretch).

I will try installing a sound card in the host machine and then reinstalling 3CX on that Server 2008R2 virtual and will see if this fixes the issues (crossing my fingers).

When I install a sound card in the host machine will the sound device be automatically installed on the virtual as a new device?

Thanks again for everyones' help. I know I can't be the only one with this issue and hopefully it can help others as well.

 

[abc123]

Most non critical devices (ie not cpu, ram, hard disk etc) are shared between all virtual machines using hyperv.

I dont think you will need to reinstall 3cx but you will probably need to reinstall the hyper v additions again for the guest os so it will correctly pick up and use the new sound card from the hypervisor host.

 

[tworhythms]

Well, I have given up...

I installed a sound card, reinstalled Server2008 (virtual machine) and 3CX and the problem persists. I even tried taking the physical machine below and converting it into a virtual for kicks. Nothing seems to fix this issue and I don't want to run my PBX on an outdated OS (i.e. Server 2003..call me an IT stickler )

I've installed 3CX on it's own dedicated Dual Core Atom machine with 2 GB Ram. It's now running very well on Server 2008R2.

Quite sad that a little Dual core Atom machine is outperforming my dual xeon server There must be some incompatibility somewhere that I'm not picking up. If I get some free time I might try loading up a test Hyper-V on another, simpler server and see what happens.

Thanks for all of your help and if anyone can think of anything else that might cause this issue, I would be more than open to try your suggestions. I am keeping the virtual machine for testing purposes at this point.

 

[abc123]

Can you play a wav file through the host machine and then the virtual machine?

Try that to make sure that it works. It may be the sound card or incompatible drivers.

 

[tworhythms]

I can play a .wav file on the host machine with no problems but I can't enable the sound device on the virtual 2008R2 machine. My understanding is sound devices can't be passed through to virtual machines. Is this correct? If they can, how do I enable this? The integrated services option doesn't include sound support as an option.

Is 3CX using the sound card? If so, many servers don't come with sound devices so how is this supported?

I have moved 3CX to the host 2008R2 server for the time being (decided that I didn't want a second PC to worry about). As before, it works flawlessly! Grr... How can 3CX run on a Dual Core Atom without problems but will not run on a Server that is hosting Exchange 2010, Blackberry Express, Domain Controller and has 2 Quad Core processors and 3 NICs? Edit- It does run on the server, just not the virtuals on that server :shock:

 

[abc123]

Sound devices dont pass through to Hyper V guests (my bad for throwing you off).

The only difference between your setup and mine is you run 2008R2 and I am on 2008 enterprise.

I have more guests running than you too.

My 3cx server is 1.5Gb ram, a virtual nic on a shared physical nic and it runs some other services for me too.

Never a problem with the audio.

One other thing to check. On your 3cx server is the Windows Audio Service running? By default on servers it is not set to automatic.

It needs to be on. On my setup, the host does not have Windows Audio Service enabled but the 3cx server does.

 

[tworhythms]

Thanks for all the suggestions.

Yes, the audio service is running (on both just to be sure). Today, I removed EVERYTHING on my network and connected the router, server and another PC (running 3CX Phone for testing) via a Gigabit switch. Still, the problem persists. It really doesn't make sense.

Also, my Hyper-V host was running DNS and Active Directory for it's own Hyper-V sub domain. I removed all of these services as well (now just a Workgroup Host). Still no luck.

I am wondering if it has something to do with the Motherboard/RAM that I'm running (ASUS Z8NA-D6 w/Non-ECC G.SKILL RAM). If it's happening on ALL of my virtuals that I load 3CX on (other than Server 2003 which still confuses me) and it doesn't seem to matter which NIC I use (I even added another to be sure).

I've also tried reinstalling NIC drivers (from ASUS) and Chipset drivers.

I think I've run out of ideas! The most frustrating/disappointing thing of it all is that it's a brand new server.

 

[comresource]

Our test 3CX system runs on a 2008R2 Hyper-V cluster with no issues. 2008R2 VM (1GB Ram), No audio card in host, no tweaking was needed. Supermicro Twin Server, 24GB per node iSCSI SAN

 

[tworhythms]

I believe I'm getting closer to narrowing this down.

Today, I put together a small Openfiler iSCSI Host and configured the exact same problematic Server 2008R2 virtual to run off VMware ESXi (vSphere) and it ran fine.

So, it seems as if this is definetely a Hyper-V configuration/driver issue. It's not the RAM/CPU/NIC or the virtual itself.

My RAID10 is much faster than a Gigabit iSCSI so if it ran fine with the iSCSI, then I think I need to look at Hyper-V.

I really like Hyper-V and want to continue using it, but if ESXi is my only solution, then I have no choice but to use it.

 

[LeonidasG_3CX]

Hi,

Have you tried the following?
http://www.3cx.com/blog/docs/voice-quality-hyperv-vmware/

 

[tworhythms]

I am using Server 2008R2 so there are no PM timers. My 2003 Server is the one that sounds good and works well. Go figure. Thanks for the suggestion though.

I am in the process of migrating all of my servers over to ESXi. It's too bad because I did like Hyper-V, but the more I get into the depths of ESXi, I like it too. Has some benefits that will help me out in the long run.

 

[sigma1]

Last night I was bored (sarcastic LOL). I tested in the lab your Hyper-V situation and worked well for us. NIC/CHIPSET/VIDEO drivers issue? I would have liked to be able to replicate the issue.

 

[tworhythms]

This is great feedback! Thanks everyone.

It's ironic that you mentioned Voip.ms since they're the provider I'm using. Oddly enough, I don't suspect them as being the direct problem as I had no issues while I was running my Server 2003 virtual.

I've tried running Server 2003 (32 Bit) and Windows 7 (32 & 64 Bit) and the only one that sounded great was the Server 2003 32 Bit. I originally suspected this as being the issue and that's when I tried it on a Win7 virtual but had no luck.

My setup is now running under VMware ESXi and is working very well with a Server 2008R2 virtual. No problems! It certainly has been a very bizzare (and frustrating) problem since I couldn't resolve it and so many others are running the same configuration without issues.

I can only surmise that the problem was with a driver (that I couldn't track down) for my NICs and the way they were handled in Hyper V. I know VMware takes a much different approach when it comes to driver provisioning and it seems as though this method "just works" with my setup.

 

[captivateglobal]

It sounds to me like the RTC (Realtime Clock) is having issues. This isn't uncommon across VMs, because RTC is based on the 1000hz battery onboard, which the computer senses when it peaks and troughs, i.e. 1000 times a second, or 1ms apart.

Now, this is easy for host computers, because it's hardware based, electricity does the work of timing for us. When you try and translate that 'ticking' into a virtual environment, it becomes unreliable, because it relies on the host tick to send back through to the virtual machine. So each time it senses an RTC tick, it ticks on the "virtual rtc" - however, because there is a small amount of (somewhat random) delay between sensing the onboard tick, and the virtual machine receiving the virtual tick, timing issues occur.

This is the same reason why it is heavily recommended that you use ntp & time syncing on virtual machines, because they are horrible at keeping time.

I know that VMWare have done extensive work on the virtual RTC and their virtualization methods seem to work pretty well, i.e. I have a VMWare machine that DOES keep time pretty well (drifts about a second a day). I haven't used Hyper-V but I would imagine it's not as well honed as VMWare in this regard, as it runs on top of a kernel which is designed to also service a host. The same is not true for ESX platforms, where there is very little happening on the host and there is less of a 'lag layer' between the physical hardware and the virtual platform.

Now when it comes to encoding audio, or at least, the timing of when to send the packets, the virtual machines if having a clock skip will send a more or less packets than is required. The reason you hear audible clicking is because an audio packet is abruptly ended - i.e. the waveform just ends, and another waveform starts. It's the same click you hear when a CD instantly skips to another part of the audio.

Now, this all works pretty well on 32-bit because the clock syncs are heavily tested in virtualization; however when you virtualize 64-bit, the I/O APIC (interrupt sharing system) latency increases slightly. This latency increase between the 'Real RTC' and the 'Virtual RTC thru I/O APIC' is most likely the cause for your 64-bit audio cracking issue. Different hardware has different support for this feature, so MMV between boards.

I've read that VMWare address this by implemention of a customized APIC for their ESX platform which gives more priority to VM over host - however, I don't think Microsoft can afford that luxury because they also need to provide the Host OS (Windows).

All I can say is I'm pretty sure it's definately got nothing to do with a NIC, VGA or CPU, and is most likely an issue with Hyper-V's APIC on specific hardware platforms.

 

[captivateglobal]

-Used Wireshark to capture and analyze SIP and RTP packets (nothing out of the normal showed up)

Just as a side note, turn on delta time, and make sure that the RTP packets sent are:

> 172 bytes on wire
> 20ms apart

If one packet arrives in -/+1ms, then there is an issue. If the packet is less than 172 bytes, it will have RTP header which specifies an 'offset' equalling the number of bytes which are missing. The packet will still be 20ms apart.

If you don't receive every packet within those specifications, you have timing issues and that will be what the clicking/popping/distorting is.