Running 3CX in a virtualized server environment

Discussion in '3CX Phone System - General' started by jswade, Sep 18, 2009.

  1. jswade

    Joined:
    Sep 18, 2009
    Messages:
    3
    Likes Received:
    0
    Just FYI for the group. I've been running the free edition of 3CX in virtualized Windows Server 2008 instance on Citrix XenServer 5.5. I don't know if there's much interest in this, but I wanted to let the community know that it can successfully be done and has the following advantages:

    - allows consolidation of platforms (3CX can run alongside other virtual servers in a consolidated environment)
    - provides the ability to create a high-availability enviromnent for 3CX (multiple physical servers can be setup in a XenServer cluster allowing 3CX to failover to another host should a physical server failure occur)
    - provides an easy way to clone/backup the 3CX server environment for disaster recovery purposes

    XenServer 5.5 and it's associated management console are free products from Citrix (high-availabilty is not one of the free features however, although manual failover is available in the free version). A few things to consider if you choose the virtualization route:

    - All gateways (PSTN, ATA, etc...) must be external (LAN attached) to the PBX. Add-in PC cards such as T1 interfaces or analog cards are not supported
    - 3CX performs best when provisioned with multiple virtual processors and set to a higher priority relative to other virtual servers that may be sharing the hardware

    If there's enough interest, I'll post a diagram of my configuration for the benefit of others.

    Thanks 3CX for a great product!

    Cheers,
    Jeff Wade
     
  2. pjr

    pjr New Member

    Joined:
    Oct 5, 2009
    Messages:
    154
    Likes Received:
    0
    Hi Jeff - this is definitely of interest so any additional info you can post would be most welcome.

    Thanks

    Paul
     
  3. jelliott52

    Joined:
    Oct 21, 2009
    Messages:
    28
    Likes Received:
    0
    Hi jswade,
    I'm running 3CX in a virtual environment as well. I'm running it with virtualbox. I have it install on a ubuntu server with virtualbox, running WinXP SP3. Works great. Actually I have another VM running as well, and instance of WinXP. This is running on a core 2 duo with 4 gigs of memory. My processors stay at about 15% usage. We are testing 3CX, so right now it's installed at my home server. but so far so good.

    Jay
     
  4. commvisi

    Joined:
    Aug 28, 2009
    Messages:
    17
    Likes Received:
    0
    HI,

    No problems here either. We run 3CX on an Windows 2008 Server x64 installed under Hyper-V. The only thing I have done differently with this VM is when I created the VHD I made it 'Fixed size' as opposed to 'Dynamically expanding' to avoid any potential Disk I/O issues. Allocated 4Gb RAM and 2 processors - sits easily below 5% most of the day.

    Troy.
     
  5. 300c_pilot

    Joined:
    Oct 26, 2009
    Messages:
    1
    Likes Received:
    0
    We are running it under Virtual Iron, it appears to work. Still addressing some issues though. Quality is not there yet.
     
  6. instantexpert

    Joined:
    Dec 2, 2008
    Messages:
    92
    Likes Received:
    0
    I'm running under VMWare Server 2 with no issues on:

    Host
    -Dual core Xenon 3GHz processor
    -8GB RAM
    -Cent OS x64 host

    Guest
    -6GB RAM
    -Windows Server 2008 SP2x32 running 3CX
    -Pre-allocated 100GB virtual machine hard disk
    -Two cores assigned to the VM

    I have no other VMs running on this machine as the virtualization was just a way to allow an easy hardware upgrade path in the future.

    On a final note, while I was making the transition, this VM resided on a VMWare Server machine with 8 cores and 16GB of memory. This machine was running several other Windows Server VMs. The audio quality was not as good when using this solution so I suspect that 3CX should be given higher priority due to its "real time" requirements.

    Leon
     
  7. darrellchapman

    Joined:
    Nov 26, 2007
    Messages:
    268
    Likes Received:
    0
    I run my 3CX environment on Windows 2003 Server Standard on top of VMWare ESX 3.5.

    Run's beautifully.
     
  8. MrKoen

    Joined:
    Jul 3, 2009
    Messages:
    60
    Likes Received:
    0
    I have 3CX running on a test machine under Microsoft Hyper-V without any problems. Host machine is running Windows 2008, the virtual host is running Windows 2003. I'm using Linksys SPA3102 and Linksys PAP2T devices to hook up PSTN lines and analog phones. The virtual image has been assigned as less as 750 MB of RAM because of it being a test machine and it still works very well.
     
  9. pjr

    pjr New Member

    Joined:
    Oct 5, 2009
    Messages:
    154
    Likes Received:
    0
    We had tested 3CX in multiple virtual server environments (2008 Hyper-V as Host OS and 2003 Server R2 as Guest OS with 3CX installed), and never had any problems, until today when we implemented 3CX into our own company environment to replace our previous IP PBX (a project on which I was the lead). It's a long explanation but well worth reading so you don't get caught out by the same thing.

    Our environment mirrored our test environment with the host server running 2008 Hyper-V and the guest OS for our 3CX server was Server 2003 R2 Std Edition. All was fine when I migrated last night but I did notice a sharp spike in CPU (100%) when McAfee updated itself at 5pm. I thought nothing much of it until I checked again this morning and noticed McAfee was still using a lot of CPU (this is with the program files and all users 3CX folders excluded from scanning). When I mentioned this to a colleague he said he would just allocate another CPU to the virtual server, which we did.

    Everything appeared to be fine but after an hour or so call quality started to degrade. This only affected voice from the 3CX server to phones, people we called could hear us fine, but it was their voice to us that was poor. LAN to LAN calls, calls to other sites across our VPN and calls via our ISDN gateway were fine. The issue only appeared to affect calls via our VoIP provider that were in effect using the 3CX server as a gateway.

    After looking at bandwidth and various other possible factors (all the while whilst having a colleague building a dedicated physical server to restore 3CX to if necessary) voice quality had gotten so bad we rebooted the server. After the reboot everything was fine again? However,sure enough over time the issue returned and calls started getting worse again.

    We had never had any problems with our previous VoIP system and the LAN and phones had not changed, only the back end PBX platform had altered. We then noticed some VERY odd behaviour on the 3CX server. Constant pings to the server from a PC on the LAN returned rock solid sub 1ms response as you would expect. Pings from the server to the same PC on the LAN would show

    <1ms
    <1ms
    <1ms
    625ms
    <1ms
    625ms
    <1ms
    <1ms
    625ms
    <1ms

    Pings from the server to it's own IP also showed the same fluctuation? We could see the behaviour of ping traffic from the server corresponded to our symptoms of poor voice being sent from the server but couldn't figure out why? None of our other virtual servers running the same OS on the same host exhibited the same behaviour? None of our testing had revealed this issue? At this stage it was 4pm and our telephone based support and sales team were none too happy with the migration as you can well imagine! It was then that a colleague of mine came across this article regarding Server 2003 Guest virtual servers using multiple CPUs

    http://techblog.mirabito.net.au/?tag=usepmtimer

    Jackpot!

    This was the only virtual server we had with multiple CPU's hence why we had not seen it before. To prove the theory we took a non critical virtual server and assigned it multiple CPUs and set up a constant ping to itself. Sure enough after 4 or so minutes ping responses to itself (that had previously been rock solid <1ms) showed odd behaviour. In fact some the of the ping responses showed negative response times!. How something can respond to a ping before it's even been sent I'm not quite sure? Either that or we've figured out time travel!?

    Following the articles recommendaion we put the \usepmtimer in the boot.ini of the guest 3CX server and rebooted and the problem seems to have disappeared. We finally got the system back to an even keel at around 5pm, by which time my nerves were shredded and Christmas bonus shrunken!

    The behaviour we saw was a result of the CPUs being in sync after a reboot but gradually drifting further and further apart.

    I hope people can learn from my experience and be sure to enter this switch if you are planning on using Server 2003 R2 as a guest OS for a multi CPU virtual machine on 2008 Hyper-V. After all the stress it was good to finally get to the root cause, and it's probably one we would have never even noticed had the platform not been running a time critical application such as 3CX, but there's got to be less stressful ways of learning these things! :lol:
     
  10. demonofsteel

    demonofsteel New Member

    Joined:
    Mar 28, 2008
    Messages:
    105
    Likes Received:
    0
    I have always run my 3CX test environments on Windows 2K3 or Windows 2K8 using VMWare and it always worked flawlessly. Much easier to work with a virtual environment than with a live PC, easier maintenance, easier to go back to a fresh install, ...

    I approve what everyone is saying, it's great!
     
  11. jelliott52

    Joined:
    Oct 21, 2009
    Messages:
    28
    Likes Received:
    0
    pjr,
    There is a couple of things I want to add and these are just MY opinions. I manage lots of VM systems and you might want to consider running Either ESX or ESXi. Windows was very late to the VM game and playing catch up. I'm a heavy linux user and I run all my windows in VM's and everybody plays nicely in the sandbox. It's when you run Window's within windows that the kids get cranky. I run VMware and Sun's virtualbox and windows loves it. I don't want to start a Win/Linux flame war here. These are just my observations and experiences.

    Jay
     
  12. DXS_Matt

    Joined:
    Feb 12, 2009
    Messages:
    41
    Likes Received:
    0
    We too run our entire infrastructure on top of the ESX platform.
    About 50 VM's between two boxes on a shared iSCSI SAN, and have had no performance issues relating to our deployment at all.

    Matt
     
  13. sbass

    Joined:
    Nov 21, 2007
    Messages:
    4
    Likes Received:
    0
    For those that stumble across pjr's post, the issue he describes isn't unique to VMs. The issue can happen on physical servers as well and it's related to systems running on AMD CPUs. The /usepmtimer switch resolves the issue with the time stamp counters and it'll fix all sorts of performance related issues. Here's a MS KB article that describes the issue: http://support.microsoft.com/kb/938448

    I ran into this on physical HP Blade servers. It has nothing to do with VMs, nor Hyper-V, nor multiple guest CPUs. It's an issue with AMD CPUs with multiple cores or multiple processors.

    Shawn
     
  14. tuxiteer

    Joined:
    Dec 26, 2009
    Messages:
    5
    Likes Received:
    0
    Have you had any issues with the IVR?
    I used to run Asterisk in a VM and had issues with users trying to get through the IVR. They complained that the voice would cut out on them and that they couldn't hear clearly. I have also had complaints about echoing (due to latency?) and repeated transmissions.
     
  15. Nick Galea

    Nick Galea Site Admin

    Joined:
    Jun 6, 2006
    Messages:
    1,677
    Likes Received:
    17
    The production system at 3Cx runs on Hyper V. we also do our bulktests on Hyper V as well as VMware ESX.

    Each release is tested on these virtual environments. If there are problems, its usually networking config or insufficient processing power or memory being allocated to that particular VM.
     
  16. nickybrg

    nickybrg Well-Known Member
    3CX Staff

    Joined:
    Jun 7, 2007
    Messages:
    1,616
    Likes Received:
    13
    Hi

    What you are experiencing is probably because of "pmtimers" on the guest.

    Check this link:

    http://wiki.3cx.com/documentation/troubleshooting/stuttering-voice-quality-in-hyperv-and-vmware

    Let us know if you have problems with this

    Thanks
     
  17. ttplastadmin

    Joined:
    Mar 5, 2010
    Messages:
    1
    Likes Received:
    0
    I am trying to replicate a setup mentioned in this thread but encountered some difficulties

    host linux ubuntu 9.10 karmic
    virtualbox 3.1.4
    guest windows xp pro 32 bit

    i cannot ping the guest nor access any services or the pbx from the network

    i would be interested in learning how those who had success in this setting have configured virtualbox;
    i am using a single network adapter in "bridged" mode; can't use NAT as the SIP provider needs to ping the pbx as well and requires ICMP responses;

    thanks a lot for your help;
     
  18. pjr

    pjr New Member

    Joined:
    Oct 5, 2009
    Messages:
    154
    Likes Received:
    0
    We have recently seen these errors on one of our 3CX virtual machines running 2003R2 on 2008 R2 Hyper-V

    The miniport 'Microsoft Virtual Machine Bus Network Adapter' hung.
    0000: 00 00 00 00 02 00 52 00 00 00 00 00 05 00 00 80
    0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    0020: 00 00 00 00 00 00 00 00

    When this happens the server drops of the network.

    Anyone else had this issue with Hyper-V?
     
  19. nickybrg

    nickybrg Well-Known Member
    3CX Staff

    Joined:
    Jun 7, 2007
    Messages:
    1,616
    Likes Received:
    13
    Check this.

    http://social.technet.microsoft.com/Forums/en/windowsserver2008r2virtualization/thread/0408a28d-6ab8-4c85-8773-4bc42c2df40b
     
  20. pjr

    pjr New Member

    Joined:
    Oct 5, 2009
    Messages:
    154
    Likes Received:
    0
    Thanks Nicky - seems to be a common issue with 2008R2 Hyper-V hosts.

    We have updated the NIC drivers and applied the hotfix in the thread and will see what happens.

    Anyone else here running 2008R2 Hyper-V as a host OS?

    Paul
     

Share This Page