Strange Hyper V Issue

Discussion in '3CX Phone System - General' started by tworhythms, Sep 15, 2010.

Thread Status:
Not open for further replies.
  1. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    I have been evaluating 3CX Version 9 for a couple of weeks now and absolutely love it!

    This is my first 3CX install so please bear with me :p

    My setup is as follows:
    XEON E5520 with 8GB Ram
    Server 2008R2 running Hyper-V on LSI Raid 1
    2 TB on dedicated LSI controller in Raid 10 (4 Sata 7200rpm drives)
    (2) Cisco SPA504G Phones as well as (2) 3CX Softphones

    All of my virtual machines are installed on the Raid 10 drives.
    My virtual machines are:
    -Domain Controller 1 GB Ram (Server 2008R2) Dedicated 10/100/1000 NIC
    -Exchange 2010 4 GB Ram (Server 2008R2) 10/100/1000 Dedicated NIC
    -3CX 2GB Ram 10/100/1000 Dedicated NIC

    When I try to run 3CX on a fresh 2008R2 Virtual machine the outgoing call quality, IVR and music on hold is fairly poor (i.e. clicking, popping and distorting).

    This is where the scenario gets strange:
    If I install 3CX on an existing Server 2003 Standard (2 GB 32-Bit version) Virtual using the same virtual switch as the 2008R2 machine and stored on the same Raid 10 setup the audio quality is great. No issues at all.

    Also, if I install 3CX directly on the Hyper-V host machine (Server 2008R2) the audio quality is excellent as well.

    So, this is what I've tried on the Virtual Server 2008R2 machine:
    -Changing the properties of the Virtual and Physical NIC (i.e. IPv4 offloading, buffers, etc...)
    -Changing to a different virtual switch
    -Implemented QoS on the Server 2008R2 virtual
    -Moved the virtual machine to a separate drive with a fixed VHD
    -Changed audio codecs, even tried using K-Lite codecs
    -Tried assigning 2 processors to the Server 2008R2 virtual instead of 1
    -Shut down Exchange 2010 and increased memory allocation to the 3CX virtual (4 GB)
    -Desktop experience and Quality Audio Video is installed
    -Changed the heartbeat, time syncronization properties of integrated services
    -Checked and double checked the firewall settings using the firewall checker
    -Used Wireshark to capture and analyze SIP and RTP packets (nothing out of the normal showed up)

    On a side note; if I install 3CX on an existing Windows 7 virtual machine using the same virtual switch (virtual NIC) the audio is terrible as well.

    So, why is the audio quality so superb while running on the Server 2003 virtual machine as opposed to the Server 2008R2 and Windows 7 virtual machines, especially considering they all use the same virtual switch (not all at the same time though)? This would lead me to believe that the problem is not with networking.

    Any help/thoughts would be truly appreciated. While I don't mind running 3CX on the Virtual Server 2003 machine, I would much rather have it running on a current OS (i.e.Server 2008R2).

    Thanks everyone!
     
  2. abc123

    abc123 Active Member

    Joined:
    Nov 9, 2009
    Messages:
    712
    Likes Received:
    1
    It appears that Win7 and Win 2k8R2 (both built using similar kernels etc.) are using a different virtual nic drivers.

    We are running 3cx on a HyperV wish 1 processor and 1.5Gb Ram but Win 2k8 Enterprise 64 and a virtual nic. No problems at all. This also runs some other apps of ours such as team foundation server etc. Load is minimal. Quality and response is high.

    Can you check your virtual nic? Do you have legacy adapter turned on in error? IF not, try turning it on to see if that makes any difference. Also double check your hyper v drivers (maybe a reinstall of the additions).

    You can also look at cpu usage on the physical machine and the 3cx virtual machine when a call is made (Check ram usage too).

    Do you have an anti virus on the 3cx machine? Is it trying to real time scan the audio?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    Thanks for the suggestions.

    Unfortunately, the issue persists. I am not running any anti-virus software on any of these virtuals until this issue is sorted out.

    Today, I tried installing a fresh Windows 7 Ultimate x64 Virtual machine and assigned 2 GB to it. The exact same problem (random clicking in the outgoing audio, IVR and music on hold). Internal calls between extensions appear to work correctly, but calls through my Voip provider have the clicking (breaking up).

    So, something doesn't make sense. Why can I make calls to the outside world with the Server 2003 virtual machine on the same host and with the same virtual switch without any issue? Ping times to my VoIP provider are very good (avg. 13ms) as I am close to one of their servers. 3CX Firewall passes all of the tests.

    The only other thing that came to mind was that the Server 2003 machine used to be a physical machine and was converted to a virtual.

    Is it necessary to have a physical sound card installed in the host machine? I wouldn't think this is the problem since I can install 3CX directly on the host machine without these audio problems.

    During calls CPU usage remains fairly low (avg 9-13 percent) and memory stays at approx. 58 percent.

    Has anybody else experienced this? I really do appreciate the help.
     
  4. robbie

    Joined:
    Sep 16, 2010
    Messages:
    1
    Likes Received:
    0
    Whe have the same problems with server 2008.
     
  5. abc123

    abc123 Active Member

    Joined:
    Nov 9, 2009
    Messages:
    712
    Likes Received:
    1
    As far as I am aware (and is the case in our setup with Hyper V) you DO need a physical sound card in the host machine.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    Update -- abc123, I think you could be on to something.

    I enabled RDP Audio passthrough on the problematic Server 2008R2 virtual machine and when I log into the machine via RDP you can see that the volume icon doesn't have the red "x" through it (i.e. RDP default audio device). Now, while I am in a RDP session and I make a call, initially the call is broken up and crackling but as soon as I start to move the volume slider on the Server 2008 virtual machine the audio is great and will remain that way for the duration of the call.

    Going back to my Server 2003 virtual (which works flawlessly), the RDP audio device ALWAYS shows enabled and is set as the default sound device. This used to be a physical machine with a sound card. Perhaps the audio drivers remained while it was converted to a virtual machine? Either that or Server 2003 handles the sound drivers completely differently (which probably isn't a stretch).

    I will try installing a sound card in the host machine and then reinstalling 3CX on that Server 2008R2 virtual and will see if this fixes the issues (crossing my fingers).

    When I install a sound card in the host machine will the sound device be automatically installed on the virtual as a new device?

    Thanks again for everyones' help. I know I can't be the only one with this issue and hopefully it can help others as well.
     
  7. abc123

    abc123 Active Member

    Joined:
    Nov 9, 2009
    Messages:
    712
    Likes Received:
    1
    Most non critical devices (ie not cpu, ram, hard disk etc) are shared between all virtual machines using hyperv.

    I dont think you will need to reinstall 3cx but you will probably need to reinstall the hyper v additions again for the guest os so it will correctly pick up and use the new sound card from the hypervisor host.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    Well, I have given up...

    I installed a sound card, reinstalled Server2008 (virtual machine) and 3CX and the problem persists. I even tried taking the physical machine below and converting it into a virtual for kicks. Nothing seems to fix this issue and I don't want to run my PBX on an outdated OS (i.e. Server 2003..call me an IT stickler :p )

    I've installed 3CX on it's own dedicated Dual Core Atom machine with 2 GB Ram. It's now running very well on Server 2008R2.

    Quite sad that a little Dual core Atom machine is outperforming my dual xeon server :( There must be some incompatibility somewhere that I'm not picking up. If I get some free time I might try loading up a test Hyper-V on another, simpler server and see what happens.

    Thanks for all of your help and if anyone can think of anything else that might cause this issue, I would be more than open to try your suggestions. I am keeping the virtual machine for testing purposes at this point.
     
  9. abc123

    abc123 Active Member

    Joined:
    Nov 9, 2009
    Messages:
    712
    Likes Received:
    1
    Can you play a wav file through the host machine and then the virtual machine?

    Try that to make sure that it works. It may be the sound card or incompatible drivers.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    I can play a .wav file on the host machine with no problems but I can't enable the sound device on the virtual 2008R2 machine. My understanding is sound devices can't be passed through to virtual machines. Is this correct? If they can, how do I enable this? The integrated services option doesn't include sound support as an option.

    Is 3CX using the sound card? If so, many servers don't come with sound devices so how is this supported?

    I have moved 3CX to the host 2008R2 server for the time being (decided that I didn't want a second PC to worry about). As before, it works flawlessly! Grr... How can 3CX run on a Dual Core Atom without problems but will not run on a Server that is hosting Exchange 2010, Blackberry Express, Domain Controller and has 2 Quad Core processors and 3 NICs? Edit- It does run on the server, just not the virtuals on that server :shock:
     
  11. abc123

    abc123 Active Member

    Joined:
    Nov 9, 2009
    Messages:
    712
    Likes Received:
    1
    Sound devices dont pass through to Hyper V guests (my bad for throwing you off).

    The only difference between your setup and mine is you run 2008R2 and I am on 2008 enterprise.

    I have more guests running than you too.

    My 3cx server is 1.5Gb ram, a virtual nic on a shared physical nic and it runs some other services for me too.

    Never a problem with the audio.

    One other thing to check. On your 3cx server is the Windows Audio Service running? By default on servers it is not set to automatic.

    It needs to be on. On my setup, the host does not have Windows Audio Service enabled but the 3cx server does.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  12. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    Thanks for all the suggestions.

    Yes, the audio service is running (on both just to be sure). Today, I removed EVERYTHING on my network and connected the router, server and another PC (running 3CX Phone for testing) via a Gigabit switch. Still, the problem persists. It really doesn't make sense.

    Also, my Hyper-V host was running DNS and Active Directory for it's own Hyper-V sub domain. I removed all of these services as well (now just a Workgroup Host). Still no luck.

    I am wondering if it has something to do with the Motherboard/RAM that I'm running (ASUS Z8NA-D6 w/Non-ECC G.SKILL RAM). If it's happening on ALL of my virtuals that I load 3CX on (other than Server 2003 which still confuses me) and it doesn't seem to matter which NIC I use (I even added another to be sure).

    I've also tried reinstalling NIC drivers (from ASUS) and Chipset drivers.

    I think I've run out of ideas! The most frustrating/disappointing thing of it all is that it's a brand new server.
     
  13. comresource

    comresource Member

    Joined:
    May 22, 2009
    Messages:
    303
    Likes Received:
    0
    Our test 3CX system runs on a 2008R2 Hyper-V cluster with no issues. 2008R2 VM (1GB Ram), No audio card in host, no tweaking was needed. Supermicro Twin Server, 24GB per node iSCSI SAN
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    I believe I'm getting closer to narrowing this down.

    Today, I put together a small Openfiler iSCSI Host and configured the exact same problematic Server 2008R2 virtual to run off VMware ESXi (vSphere) and it ran fine.

    So, it seems as if this is definetely a Hyper-V configuration/driver issue. It's not the RAM/CPU/NIC or the virtual itself.

    My RAID10 is much faster than a Gigabit iSCSI so if it ran fine with the iSCSI, then I think I need to look at Hyper-V.

    I really like Hyper-V and want to continue using it, but if ESXi is my only solution, then I have no choice but to use it.
     
  15. LeonidasG

    LeonidasG Support Team
    Staff Member 3CX Support

    Joined:
    Nov 19, 2008
    Messages:
    1,455
    Likes Received:
    92
    Hi,

    Have you tried the following?
    http://www.3cx.com/blog/docs/voice-quality-hyperv-vmware/
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    I am using Server 2008R2 so there are no PM timers. My 2003 Server is the one that sounds good and works well. Go figure. Thanks for the suggestion though.

    I am in the process of migrating all of my servers over to ESXi. It's too bad because I did like Hyper-V, but the more I get into the depths of ESXi, I like it too. Has some benefits that will help me out in the long run.
     
  17. sigma1

    sigma1 Active Member

    Joined:
    Nov 20, 2009
    Messages:
    542
    Likes Received:
    1
    Last night I was bored (sarcastic LOL). I tested in the lab your Hyper-V situation and worked well for us. NIC/CHIPSET/VIDEO drivers issue? I would have liked to be able to replicate the issue.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    176
    Likes Received:
    14
    I'm running Hyper-V on Server 2008 R2 on a Dell PowerEdge 2900.

    For 3CX, I set up a 32-bit Windows 7 Ultimate guest with 1 processor and 1GB of RAM. I initially had a bit of clicking with the VoIP provider (voip.ms) but not with a PSTN gateway. I haven't even heard the clicking lately.

    The host does not have a sound card. The Audio service is not running. Device Manager lists no "Sound, video and game controllers" and under System Devices, there is no System Speaker listed.

    The Win7 guest shows Audio Service running and speaker at 100% when I log on. Device Manager lists no "Sound, video and game controllers" but under System Devices, the System Speaker is there.

    I turned off Time Sync for ALL VM guests and use standard domain time sync after getting time errors (blogged).

    Have you tried a 32-bit guest?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. tworhythms

    Joined:
    Sep 15, 2010
    Messages:
    19
    Likes Received:
    0
    This is great feedback! Thanks everyone.

    It's ironic that you mentioned Voip.ms since they're the provider I'm using. Oddly enough, I don't suspect them as being the direct problem as I had no issues while I was running my Server 2003 virtual.

    I've tried running Server 2003 (32 Bit) and Windows 7 (32 & 64 Bit) and the only one that sounded great was the Server 2003 32 Bit. I originally suspected this as being the issue and that's when I tried it on a Win7 virtual but had no luck.

    My setup is now running under VMware ESXi and is working very well with a Server 2008R2 virtual. No problems! It certainly has been a very bizzare (and frustrating) problem since I couldn't resolve it and so many others are running the same configuration without issues.

    I can only surmise that the problem was with a driver (that I couldn't track down) for my NICs and the way they were handled in Hyper V. I know VMware takes a much different approach when it comes to driver provisioning and it seems as though this method "just works" with my setup.
     
  20. mcbsys

    mcbsys New Member

    Joined:
    Oct 8, 2008
    Messages:
    176
    Likes Received:
    14
    If your NIC is WHQL-certified I supposed you could raise a Microsoft support incident. Sounds like you already spent way more time on this than reasonable, though! Glad ESXi is working.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Thread Status:
Not open for further replies.