3CX Premium Partner, HaloKwadrat, carried out a series of extensive performance tests on 3CX Phone System. Here are their results.
3CX Phone System is one of the most popular IP PBX systems that works flawlessly in a Microsoft Windows environment. It’s renowned for its simplicity, scalability and also how easy it is to configure. 3CX licensing is primarily based on limiting the number of simultaneous calls that can be made. However, we have found that many customers look on 3CX with some disbelief at the sensibility of the 512 SC (simultaneous calls) license as 3CX is perceived more as an SMB phone system solution. Therefore, what happens when we test the system to its licensing maximum?
We would like to thank 3CX for making the 512 SC Enterprise Edition available for our tests, especially since we made it explicit that we were not going to be lenient about any unfavorable findings. Our aim was to rigorously test 3CX to the limit; therefore we were not going to do a simple test using a basic SIP generator with 512 simultaneous calls running. We were going to go one step further…
3CX Phone System Thesis
We wanted to check how 3CX Phone System will perform in a simulated office environment of a large enterprise, so we went to work and:
- introduced 3,000 internal extensions
- introduced a complex PBX configuration
- created a SIP generator and terminator that emulated different types of traffic and media
- emulated a SIP trunk to PSTN
- created a configuration in which 3CX routed the traffic using DDIs, ACD and SIP Location Service
- created a recording configuration and made sure that incoming calls are using IVR
We were also going to receive a recording of the system load, however we needed a reference load in the context of the operating system overhead and so we went in the direction of virtualization.
Apart from some testing aspects, we decided that virtualization is one of the growing types of IP PBX implementations and generally used in an enterprise IT environment. We wanted to see whether using a hypervisor would cause issues in any way with real time communications when at a high load level.
1.0 Test Procedure
1.1 3CX Environment
A Dell PowerEdge was to host the hypervisor, VMware, and 3CX Phone System software:
|Model||Dell PowerEdge 1950 MKII|
|CPU||2x Intel Xeon E5320 @ 1,86 GHz|
|RAM||16 GB DDR2-667 PC2-5300|
|Hard Drive||2×300 GB SAS 15k|
|RAID||PERC 5/I running RAID1|
|NIC||2x Broadcom 1000 Base-T|
|Model||VMware ESXi 4.1.0|
1.1.3 Installed VM
We decided that in order to simulate a working environment, we needed to create two more virtual machines apart from the one that would be tested. The VMs were in idle state and, apart from OS systems, no services were active.
|VM||3CX 512 Test||VM1||VM2|
|Description||VM for 3CX||Idle VM1||Idle VM2|
|CPU||4 vCPU (4 Cores from Xeon CPU)||2 vCPU (2 Cores from Xeon CPU)||2 vCPU (2 Cores from Xeon CPU)|
|Hard Drive||83GB SCSI (0:0) datastore1||22GB SCSI (0:0) datastore1||22GB SCSI (0:0) datastore1|
|RAID||LSI Logic SAS||LSI Logic Parallel||LSI Logis SAS|
|NIC||1x E1000||2x E1000 (Different vSwitch)||2xE1000 (Different vSwitch)|
|OS||Windows Server 2008 R2 Standard||Debian Squeeze 6.0.2||Centos 6.0|
18.104.22.168 3CX Software on VM
The following software was installed on the VM with 3CX Phone System on board. In order to catch reference performance with Microsoft software we also used the option of ‘Report Performance’ from the VMware vSphere Client 4.1.0.
|Operating System||Microsoft Windows Server 2008 R2 Standard|
|IP PBX||3CX Phone System 10, Enterprise Edition 512SC, Service Pack 2|
|OS Performance Monitor||Microsoft Performance Monitor|
|VM Performance Monitor||VMware vSphere Client 4.1.0|
1.2 Testing / Application Environment
For the SIP generators and terminators of the various types of traffic, there were two machines with approximately the following parameters:
|CPU||1x Intel QC @ 2GHz|
|RAM||4 GB DDR3|
|Hard Drive||1x 1TB SATAII|
|NIC||1x Intel Pro E1000 1000 Base-T|
The SIP generators and terminators used the following software configuration:
|Emulator / Generator SIP||SIPp v3.2-TLS-PCAP|
|RTP Generator Library||libpcap 1.0.0|
|SSL Generator||openssl 1.0.0|
|HTTP Generator||Curl-loader 0.53|
1.3 Testing Scenarios
1.3.1 Voice Scenarios
We differentiated the following “actors” in the test procedures:
- Extension Voice – internal extension that generates audio
- Extension – internal extension registered to the PBX without generating audio
- SIP Trunk – simulating access to PSTN
- We simulated the following traffic types:
- IVR – incoming traffic from SIP trunk to DDI routed via IVR (DTMF) and directed to an ACD queue from which it directed to Extension Voice
- IN – incoming traffic directed from SIP Trunk to DDI and directly to Extension Voice
- OUT – outgoing voice traffic generated from Extension Voice and directed to the SIP Trunk.
- LOCAL – voice traffic generated by Extension Voice and directed to other Extension Voice
In the table below we have specified all types of traffic together with the maximum number of simultaneous calls of each traffic type. For some types of traffic we have defined specific scenarios such as: call recording, Automatic Call Distribution based on a Round-Robin algorithm as well as using the PBX as an RTP / media proxy.
|Traffic Type||Initiator||Terminator||Recording||ACD||Media Proxy||Calls|
|IVR-NOREC||SIP Trunk||Extension Voice||NO||YES||YES||82|
|IVR-REC||SIP Trunk||Extension Voice||YES||YES||YES||20|
|IN-NOREC||SIP Trunk||Extension Voice||NO||NO||YES||189|
|IN-REC||SIP Trunk||Extension Voice||NO||NO||YES||21|
|OUT||Extension Voice||SIP Trunk||N/A||N/A||NO||90|
|LOCAL||Extension Voice||Extension Voice||N/A||N/A||NO||110|
1.3.2 SIP Registration Generator
Every Extension Voice had to be registered as a SIP UA to the PBX so it could initiate or terminate a voice connection. The other 2378 extensions also had to register as SIP UA irrespective of the fact that they did not generate voice traffic.
|REG-VOICE||Extension Voice||3CX Location Service||Digest Authentication||622|
|IVR-REC||Extension||3CX Location Service||Digest Authentication||2378|
1.3.3 HTTP Queries Generator
The Admin and user GUI is based on Abyss WWW server from Aprelium. Our last test scenario component was generating HTTP GET queries to the WWW server as an approximation of how many users would want to check missed calls, for example, or reconfigure their follow me rules.
|HTTP GET||curl-loader||Abyss Web Server||3CX MyPhone Login Page||25/Sec|
1.4 Test Method
We specified 3 test methods, from which we would analyze results.
1.4.1 Idle Status
In this method we wanted to find out the reference load on the server when it does not process any SIP connections.
1.4.2 8 Calls per 10 Seconds
In this method we decided to generate 8 calls every 10 seconds, where every call had a length of 800 seconds. We wanted to see how the system performed when the load was rising at a linear pace and test the system resilience to a high load over a long period of time. Simultaneously, we were generating 25 queries per second to the WWW server, which is part of the 3CX Phone System application.
1.4.3 8 Calls per Second
In this method we decided to generate 8 calls every second, where every call was 80 seconds long. This scenario is unlikely in the real world but we wanted to see how the system performed when the load was rising fast and test the system resilience to a high resource allocation in a short period of time. Simultaneously we were generating 25 queries per second to the WWW server, which is part of the 3CX Phone System application.
1.5 Collecting Results
We used Microsoft Performance Monitor as well as VMware vSphere Client to CSV files. We were interested in: CPU Load, Memory usage, I/O Average, Network Bandwidth Average, CPU Time for 3CX Processes and Datastore Latency.
While preparing the test environment, we encountered a few elements that need highlighting to better understand the overall test and results.
2.1 SIP Generator
In order to obtain an appropriate packet sequence that simulated SIP registration, call processing, authorization and call disconnect, we needed a very scalable SIP generator. We chose Sipp, an open source tool that has been used for years in various SIP stress tests. The main reason for this choice is that it has low hardware requirements and is simple to use.
The only drawback, which we encountered while using Sipp, was that when under a high load, 3CX Phone System would occasionally send SIP messages in the wrong sequence. This generated errors in the next Sipp call attempts. The structure of each test assumes a rigorous SIP message sequence. Our workaround was to treat some SIP messages as optional.
2.2 RTP Generator
The biggest challenge of testing VoIP solutions is the necessity to generate a large number of RTP streams in an appropriate codec. Since coding such a high number of RTP streams would require a processor or a transcoding card, we decided to use a Sipp module called PCAP Play. PCAP Play allowed us to play an 8 second sound file transmitted using G711A and loop it the required number of times in order to achieve desired audio sample length.
2.3 SIP/RTP Terminator
In the case of the SIP/RTP Terminator we used the same tricks as in the Generator. The RTP stream was echoed back to Generator using the Sipp RTP Echo application.
2.4 3CX Phone System Configuration
3CX Phone System was configured with 3,000 user accounts and appropriate routing rules. Thankfully, 3CX Phone System allows a simple import of a CSV files. This cut the required time from years to 15 minutes, and all necessary info was quickly imported.
Creating DDIs proved to be a problem since 3CX Phone System does not offer an automatic import of DDIs. Inserting them manually was a pain.
The graphs below present the resources usage for OS and 3CX Phone System Enterprise 512SC Edition while no calls were being processed and no SIP or HTTP queries were executed.
3.1.1 Microsoft Performance Monitor
The results below were collected using Microsoft Performance Monitor and present the total usage of I/O hard drive (read/write) in bytes per second, CPU usage time (4 cores) in a percentage scale, available (not allocated) RAM memory in Megabytes, as well as the total bandwidth transferred through a network interface (Rx / Tx) in Bytes per second. All other results will be presented in the same fashion.
On the graphs below we see the total CPU usage for individual processes that are part of 3CX Phone System, while no connections (SIP or HTTP) were made.
3.1.2 VMware vSphere
Below, we see the results collected using VMware vSphere Client Performance Report which presents the total I/O usage of physical hard drive (read/write) in kilobytes per second, total usage of CPU (all 4 cores) in a percentage scale, and used RAM in kilobytes per second. Other tests from vSphere will be presented in the same fashion for other test scenarios.
3.2 8 Calls per 10s
The graphs below present resource usage while conducting the 8 calls per 10 seconds test scenario. In these graphs we have included the number of active calls. The test lasted for 25 minutes. The final peak in I/O results is a consequence of an RDP access login.
The presented results using vSphere and PerfMon show very similar characteristics of resource usage. This simply means that resources are correctly allocated and freed.
While using this test, we had 3% lost calls in the ACD mechanism or due to a wrong SIP message sequence sent to SIP generator. This could also be a timing issue between virtual machines. On the terminator side we did not notice any problems.
It is noticeable that bigger resources are needed for call setups then for the maintenance of the call. This is not applicable to I/O HDD which, after call setup and IVR playback, had to record some of the calls.
3.2.1 Microsoft Performance Monitor
Below are the results of this test using Microsoft Performance Monitor.
On the graph below one can observe the total CPU usage for individual processes of 3CX Phone System at a call rate of 8 calls per second.
3.2.2 VMware vSphere
Below are the results of this test, collected using VMware vSphere Client Performance Report.
3.3 8 Call per Second – Stress Test
The graphs below present the resource usage while conducting the 8 calls per second test scenario. In these graphs we have included the number of active calls.
The presented results, using vSphere and PerfMon, show very similar characteristics of resource usage. This simply means that resources are correctly allocated and freed.
In this test, failed calls amounted to 8%. This is because of the large number of calls being setup simultaneously. Such scenarios are extremely unrealistic in real world production environments. Again, the errors were caused by calls being lost in the ACD engine and the wrong sequencing of SIP messages sent to SIP generator.
We are aware that these 37 calls were dropped because of a lack of available CPU resources in the time of 13:20 peak.
Again it is noticeable that system resources needed are higher for call setups rather then call maintenance. This does not apply to I/O HDD which behaved differently to the first test and recorded highest load at the end of the test. This is due to recording of calls.
3.3.1 Microsoft Performance Monitor
Below are the results of this test collected using VMware vSphere Client Performance Report.
On the graph below one can observe that the total CPU usage for individual processes of 3CX application while the 8 calls per second test was conducted.
3.3.2 VMware vSphere
Below are the results of this test collected using VMware vSphere Client Performance Report.
A system that can handle advanced call scenarios for 512 simultaneous calls can easily handle a company with up to 3,000 extensions. Such a number is akin to a small ITSP. This means that 3CX has designed 3CX Phone System very ambitiously indeed.
Our tests were conducted with a 50/50 feeling, especially given the demanding scenarios designed (virtualization, recording, abnormally high number of call setups per second). However our tests have shown that the 512 SC license is not just a marketing gimmick and 3CX Phone System is not only an SMB product but, with appropriate architecture design, has potential for even large enterprises.
While examining these results we also bear in mind that the license cost and maintenance cost of 3CX Phone System is probably the lowest available on the market.
Please note: Adding more resources to the machine, for example, more RAM & more CPU power, would resolve the failed calls issue we saw when pushing to the limit.
About the Authors
The tests on 3CX Phone System were carried out by Michał Podoski and Andrzej Pierścionek.
Michal Podoski has been active in the IT and Telecommunication market for over 7 years. During that time, he held senior engineering and managerial positions. He is a carrier grade class specialist in networking and VoIP services. For many years, Michał has been using and abusing the potential of open source voice platforms. His main interests are securing in SIP, VAS, VoIP and cloud computing networks. He is currently a senior engineer at HaloKwadrat.
Andrzej Pierscionek has gained experience in implementing software PBX infrastructure and creating call centre solutions. His main interests are networks of all sizes and VoIP. Andrzej is currently an engineer at HaloKwadrat.
HaloKwadrat is a 3CX Distributor in Warsaw, Poland and prides itself on being a leading distributor in the field of VoIP telephony. HaloKwadrat distinguishes itself by combining excellent pricing along with its great support of 3CX Phone System, Patton, Sangoma, and Polycom products. HaloKwadrat organizes basic and advanced 3CX training events each year and works with over 200 partners throughout Europe.