• V20: 3CX Re-engineered. Get V20 for increased security, better call management, a new admin console and Windows softphone. Learn More.

Running 3CX in a virtualized server environment

Status
Not open for further replies.

jswade

Joined
Sep 18, 2009
Messages
3
Reaction score
0
Just FYI for the group. I've been running the free edition of 3CX in virtualized Windows Server 2008 instance on Citrix XenServer 5.5. I don't know if there's much interest in this, but I wanted to let the community know that it can successfully be done and has the following advantages:

- allows consolidation of platforms (3CX can run alongside other virtual servers in a consolidated environment)
- provides the ability to create a high-availability enviromnent for 3CX (multiple physical servers can be setup in a XenServer cluster allowing 3CX to failover to another host should a physical server failure occur)
- provides an easy way to clone/backup the 3CX server environment for disaster recovery purposes

XenServer 5.5 and it's associated management console are free products from Citrix (high-availabilty is not one of the free features however, although manual failover is available in the free version). A few things to consider if you choose the virtualization route:

- All gateways (PSTN, ATA, etc...) must be external (LAN attached) to the PBX. Add-in PC cards such as T1 interfaces or analog cards are not supported
- 3CX performs best when provisioned with multiple virtual processors and set to a higher priority relative to other virtual servers that may be sharing the hardware

If there's enough interest, I'll post a diagram of my configuration for the benefit of others.

Thanks 3CX for a great product!

Cheers,
Jeff Wade
 
Hi Jeff - this is definitely of interest so any additional info you can post would be most welcome.

Thanks

Paul
 
Hi jswade,
I'm running 3CX in a virtual environment as well. I'm running it with virtualbox. I have it install on a ubuntu server with virtualbox, running WinXP SP3. Works great. Actually I have another VM running as well, and instance of WinXP. This is running on a core 2 duo with 4 gigs of memory. My processors stay at about 15% usage. We are testing 3CX, so right now it's installed at my home server. but so far so good.

Jay
 
HI,

No problems here either. We run 3CX on an Windows 2008 Server x64 installed under Hyper-V. The only thing I have done differently with this VM is when I created the VHD I made it 'Fixed size' as opposed to 'Dynamically expanding' to avoid any potential Disk I/O issues. Allocated 4Gb RAM and 2 processors - sits easily below 5% most of the day.

Troy.
 
We are running it under Virtual Iron, it appears to work. Still addressing some issues though. Quality is not there yet.
 
I'm running under VMWare Server 2 with no issues on:

Host
-Dual core Xenon 3GHz processor
-8GB RAM
-Cent OS x64 host

Guest
-6GB RAM
-Windows Server 2008 SP2x32 running 3CX
-Pre-allocated 100GB virtual machine hard disk
-Two cores assigned to the VM

I have no other VMs running on this machine as the virtualization was just a way to allow an easy hardware upgrade path in the future.

On a final note, while I was making the transition, this VM resided on a VMWare Server machine with 8 cores and 16GB of memory. This machine was running several other Windows Server VMs. The audio quality was not as good when using this solution so I suspect that 3CX should be given higher priority due to its "real time" requirements.

Leon
 
I run my 3CX environment on Windows 2003 Server Standard on top of VMWare ESX 3.5.

Run's beautifully.
 
I have 3CX running on a test machine under Microsoft Hyper-V without any problems. Host machine is running Windows 2008, the virtual host is running Windows 2003. I'm using Linksys SPA3102 and Linksys PAP2T devices to hook up PSTN lines and analog phones. The virtual image has been assigned as less as 750 MB of RAM because of it being a test machine and it still works very well.
 
We had tested 3CX in multiple virtual server environments (2008 Hyper-V as Host OS and 2003 Server R2 as Guest OS with 3CX installed), and never had any problems, until today when we implemented 3CX into our own company environment to replace our previous IP PBX (a project on which I was the lead). It's a long explanation but well worth reading so you don't get caught out by the same thing.

Our environment mirrored our test environment with the host server running 2008 Hyper-V and the guest OS for our 3CX server was Server 2003 R2 Std Edition. All was fine when I migrated last night but I did notice a sharp spike in CPU (100%) when McAfee updated itself at 5pm. I thought nothing much of it until I checked again this morning and noticed McAfee was still using a lot of CPU (this is with the program files and all users 3CX folders excluded from scanning). When I mentioned this to a colleague he said he would just allocate another CPU to the virtual server, which we did.

Everything appeared to be fine but after an hour or so call quality started to degrade. This only affected voice from the 3CX server to phones, people we called could hear us fine, but it was their voice to us that was poor. LAN to LAN calls, calls to other sites across our VPN and calls via our ISDN gateway were fine. The issue only appeared to affect calls via our VoIP provider that were in effect using the 3CX server as a gateway.

After looking at bandwidth and various other possible factors (all the while whilst having a colleague building a dedicated physical server to restore 3CX to if necessary) voice quality had gotten so bad we rebooted the server. After the reboot everything was fine again? However,sure enough over time the issue returned and calls started getting worse again.

We had never had any problems with our previous VoIP system and the LAN and phones had not changed, only the back end PBX platform had altered. We then noticed some VERY odd behaviour on the 3CX server. Constant pings to the server from a PC on the LAN returned rock solid sub 1ms response as you would expect. Pings from the server to the same PC on the LAN would show

<1ms
<1ms
<1ms
625ms
<1ms
625ms
<1ms
<1ms
625ms
<1ms

Pings from the server to it's own IP also showed the same fluctuation? We could see the behaviour of ping traffic from the server corresponded to our symptoms of poor voice being sent from the server but couldn't figure out why? None of our other virtual servers running the same OS on the same host exhibited the same behaviour? None of our testing had revealed this issue? At this stage it was 4pm and our telephone based support and sales team were none too happy with the migration as you can well imagine! It was then that a colleague of mine came across this article regarding Server 2003 Guest virtual servers using multiple CPUs

http://techblog.mirabito.net.au/?tag=usepmtimer

The symptoms were that my Windows Server 2003 machine would return very strange results when pinging hosts, both internally and externally, such as returning all four responses within about half a second, yet measuring them at over 3000ms (which means they should have timed out, rather than given me a reading in milliseconds) as well as occasionally providing negative values for response times.

Obviously the results were completely inaccurate, but I couldn’t work out why the issue was only happening on a handful (not all) Hyper-V guests running Windows Server 2003 and none on Server 2008.

Turns out that this is an issue if all of the following are true:

You are running an operating system prior to Windows Vista or Windows Server 2008
You are running the current implementation of Microsoft Hyper-V (i.e. at the time of writing)
You have presented multiple processors to the Hyper-V guest

The issue occurs because the multiprocessor HAL in Hyper-V causes the guest’s operating system Time Stamp Counter (TSC) to skew. According to this blog the problem wouldn’t ordinarily occur if you were running Windows Server 2003 with SP2 unless the BIOS check fails to determine if the TSC should be used. More specifically, if I understand correctly the issue occurs because the processors (or cores, if we’re talking about a single multicore processor) are not in sync with each other, which produces sporadic out-of-time results where time sensitive operations (such as ping responses) are in use.

Jackpot!

This was the only virtual server we had with multiple CPU's hence why we had not seen it before. To prove the theory we took a non critical virtual server and assigned it multiple CPUs and set up a constant ping to itself. Sure enough after 4 or so minutes ping responses to itself (that had previously been rock solid <1ms) showed odd behaviour. In fact some the of the ping responses showed negative response times!. How something can respond to a ping before it's even been sent I'm not quite sure? Either that or we've figured out time travel!?

Following the articles recommendaion we put the \usepmtimer in the boot.ini of the guest 3CX server and rebooted and the problem seems to have disappeared. We finally got the system back to an even keel at around 5pm, by which time my nerves were shredded and Christmas bonus shrunken!

The behaviour we saw was a result of the CPUs being in sync after a reboot but gradually drifting further and further apart.

I hope people can learn from my experience and be sure to enter this switch if you are planning on using Server 2003 R2 as a guest OS for a multi CPU virtual machine on 2008 Hyper-V. After all the stress it was good to finally get to the root cause, and it's probably one we would have never even noticed had the platform not been running a time critical application such as 3CX, but there's got to be less stressful ways of learning these things! :lol:
 
I have always run my 3CX test environments on Windows 2K3 or Windows 2K8 using VMWare and it always worked flawlessly. Much easier to work with a virtual environment than with a live PC, easier maintenance, easier to go back to a fresh install, ...

I approve what everyone is saying, it's great!
 
pjr,
There is a couple of things I want to add and these are just MY opinions. I manage lots of VM systems and you might want to consider running Either ESX or ESXi. Windows was very late to the VM game and playing catch up. I'm a heavy linux user and I run all my windows in VM's and everybody plays nicely in the sandbox. It's when you run Window's within windows that the kids get cranky. I run VMware and Sun's virtualbox and windows loves it. I don't want to start a Win/Linux flame war here. These are just my observations and experiences.

Jay
 
We too run our entire infrastructure on top of the ESX platform.
About 50 VM's between two boxes on a shared iSCSI SAN, and have had no performance issues relating to our deployment at all.

Matt
 
For those that stumble across pjr's post, the issue he describes isn't unique to VMs. The issue can happen on physical servers as well and it's related to systems running on AMD CPUs. The /usepmtimer switch resolves the issue with the time stamp counters and it'll fix all sorts of performance related issues. Here's a MS KB article that describes the issue: http://support.microsoft.com/kb/938448

I ran into this on physical HP Blade servers. It has nothing to do with VMs, nor Hyper-V, nor multiple guest CPUs. It's an issue with AMD CPUs with multiple cores or multiple processors.

Shawn
 
Have you had any issues with the IVR?
I used to run Asterisk in a VM and had issues with users trying to get through the IVR. They complained that the voice would cut out on them and that they couldn't hear clearly. I have also had complaints about echoing (due to latency?) and repeated transmissions.
 
The production system at 3Cx runs on Hyper V. we also do our bulktests on Hyper V as well as VMware ESX.

Each release is tested on these virtual environments. If there are problems, its usually networking config or insufficient processing power or memory being allocated to that particular VM.
 
Hi

What you are experiencing is probably because of "pmtimers" on the guest.

Check this link:

http://wiki.3cx.com/documentation/troubleshooting/stuttering-voice-quality-in-hyperv-and-vmware

Let us know if you have problems with this

Thanks
 
I am trying to replicate a setup mentioned in this thread but encountered some difficulties

host linux ubuntu 9.10 karmic
virtualbox 3.1.4
guest windows xp pro 32 bit

i cannot ping the guest nor access any services or the pbx from the network

i would be interested in learning how those who had success in this setting have configured virtualbox;
i am using a single network adapter in "bridged" mode; can't use NAT as the SIP provider needs to ping the pbx as well and requires ICMP responses;

thanks a lot for your help;
 
We have recently seen these errors on one of our 3CX virtual machines running 2003R2 on 2008 R2 Hyper-V

The miniport 'Microsoft Virtual Machine Bus Network Adapter' hung.
0000: 00 00 00 00 02 00 52 00 00 00 00 00 05 00 00 80
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0020: 00 00 00 00 00 00 00 00

When this happens the server drops of the network.

Anyone else had this issue with Hyper-V?
 
Check this.

http://social.technet.microsoft.com/Forums/en/windowsserver2008r2virtualization/thread/0408a28d-6ab8-4c85-8773-4bc42c2df40b
 
Thanks Nicky - seems to be a common issue with 2008R2 Hyper-V hosts.

We have updated the NIC drivers and applied the hotfix in the thread and will see what happens.

Anyone else here running 2008R2 Hyper-V as a host OS?

Paul
 
Status
Not open for further replies.

Getting Started - Admin

Latest Posts

Forum statistics

Threads
141,632
Messages
748,962
Members
144,747
Latest member
Murad88
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.