3cx on High Availability

[inss.3.c.x]

Hello,

What is the best way to create a High Availability cluster for 3CX? There are a couple articles in the documentation referring to HA, Failover and SBC, but i'm not quite clear what the best route is.

Is active/active possible or only active-passive failover? The articles don't provide too many details except for a basic instructions set. What are the prerequisites and elements we need ?

Is it possible to build it with custom FQDNs and SSL certs? I really don't want to rely on a 3rd party for my domain names and SSL certs, although other opinions say they prefer to have 3CX do that for them.

Once it is in a cluster, what should we expect? how would I recover the primary server and failback? What is the correct procedure for manual failover and failback?

Does it truly work? is it really worth the trouble? Or should I stick to a cold or warm standby system?

 

[inss.3.c.x]

Also what happen with the recordings in a 3CX cluster? does 3CX replicates the call recording folder to the same location on the backup server? are they kept in sync real time ?

 

[cobaltit]

What documents have you been looking at? This one covers it pretty well and answers all of your questions:

https://www.3cx.com/docs/failover/

 

[Evolute IT]

What is the best way to create a High Availability cluster for 3CX? There are a couple articles in the documentation referring to HA, Failover and SBC, but i'm not quite clear what the best route is.

Is active/active possible or only active-passive failover? The articles don't provide too many details except for a basic instructions set. What are the prerequisites and elements we need ?

So, if you want the best reliability, the 3CX Failover is the way to go. It takes care of everything, up until DNS (if you use the 3CX FQDN.)

after that, the SBC HA can help render it truely reliable but still, the SBC itself is almost unkillable. The thing is made to be solid and is tested as such by 3CX themselves.

You can either have 2 systems with a Pro license each, or have 1 Enterprise license for both, since Ent includes a stand-by license for the passive server.

While you can't do active-active, the failover takes place usually rapidly and Ent licenses have a TTL of 5min on the DNS. Pro has 6h. Meaning, Enterprise is much faster to switch to the passive one.

Is it possible to build it with custom FQDNs and SSL certs? I really don't want to rely on a 3rd party for my domain names and SSL certs, although other opinions say they prefer to have 3CX do that for them.

Yes, a Pro or Enterprise license will give you this ability. However, you need to manually renew and replace the certificate files when due and failover will require a custom Bash or Batch script to update your own DNS records to the new IP (usually done when using local WinServer DNS.)

Once it is in a cluster, what should we expect? how would I recover the primary server and failback? What is the correct procedure for manual failover and failback?

Normally, you would shut down the passive one (in a scheduled maintenance window), restore a new VM using the latest backup and set it as Active failover. When booting, the passive one will detect that and not start services. There's no manual failover possible since it works by monitoring a hearbeat from the active one. You can just shutdown the active server and the passive will take over.

Does it truly work? is it really worth the trouble? Or should I stick to a cold or warm standby system?

It does. It's really great! If you use VMware with vMotion or a Hyper-V cluster, it is not really needed. But for the Cloud, it is great.

Also what happen with the recordings in a 3CX cluster? does 3CX replicates the call recording folder to the same location on the backup server? are they kept in sync real time ?

Usually, in this case, you would keep recordings and backups stored on a Google Bucket or SFTP/SMB. This way, when the passive does its scheduled restore, it has access to the files. It also reduces the size of the backup file to make it fast and efficient. (The passive does a scheduled restore of the latest backup in order to take over with the latest backed up data when needed.)

All that being said, if you need a 3CX Partner just to help you get started, feel free to DM me! Also, we can provide you with a PoC license to test the system and it's features.

 

[inss.3.c.x]

Thank you both!

So if i were to use AWS, I'd do an EBS volume say 30GB for the OS and 3CX and a separate volume maybe 500GB for the recordings.

According to what you explained about the recordings, I could use EFS for that second module since i can attach it to multiple EC2 instances at a time, and set it up with auto growth. Then both primary and secondary 3CX would share the same data volume for the recordings.

Then I could do the backups of the system config only, and backup the recordings separately either via a snapshot of "recordings" volume, or do linux rsync to yet a 2nd EFS volume, or aws sync to a private S3 bucket.

Would that be a feasible scenario? I believe none of this would be supported by 3CX anyway since it steers away from their standard straight forward configurations.

Regarding the documentation, these are the articles i'm reading:

https://www.3cx.com/blog/releases/sbc-high-availability-clustering/
https://www.3cx.com/docs/failover/
https://www.3cx.com/docs/sbc-high-availability-cluster/
https://www.3cx.com/community/threads/3cx-failover-or-ha-in-enterprise-edition.53674/

And reviewing the hardware requirements, 3CX recommends a r5ad.xlarge machine in AWS for a 250+ extensions system. That's 4 vCPUs and 32GB RAM

What would the performance be like for a 3CX system with 200 extensions, 64 simultaneous calls, IVRs, multiple queues, call recording, etc and a CRM integration, under that VM config? Would/Should it be enough (in average) or should I go a bit bigger?

 

[Evolute IT]

So if i were to use AWS, I'd do an EBS volume say 30GB for the OS and 3CX and a separate volume maybe 500GB for the recordings.

According to what you explained about the recordings, I could use EFS for that second module since i can attach it to multiple EC2 instances at a time, and set it up with auto growth. Then both primary and secondary 3CX would share the same data volume for the recordings.

Then I could do the backups of the system config only, and backup the recordings separately either via a snapshot of "recordings" volume, or do linux rsync to yet a 2nd EFS volume, or aws sync to a private S3 bucket.

I would personally recommend going the SFTP or Google Bucket in your case. The supported installations are not modified, meaning mounting the second EBS volume would render your install unsupported. Also, I am unsure what the results would be. I don't think anyone ever thought of going that way! I'd be happy to help you with this and figure out if this can actually work (it would help us also lol.)

Using an external storage supported by 3CX has the advantage of being assured to work.

And reviewing the hardware requirements, 3CX recommends a r5ad.xlarge machine in AWS for a 250+ extensions system. That's 4 vCPUs and 32GB RAM

What would the performance be like for a 3CX system with 200 extensions, 64 simultaneous calls, IVRs, multiple queues, call recording, etc and a CRM integration, under that VM config? Would/Should it be enough (in average) or should I go a bit bigger?

Since you plan on using call recording and the integration and all, I strongly recommend going 8 cores/32GB RAM. This way, the CPU has plenty of Hz to encode audio.

This sounds like a great setup you will have there! I'd be glad to assist in anyway I can (we are a 3CX Partner )

 

[inss.3.c.x]

Thank you very much for all the help and the info, I really appreciate it. I'm using AWS free tear right now for a PoC, basically to understand how 3cx behaves under different conditions. I got the 1 year free license.

You mentioned SFTP or Google bucket. If i read the documentation correctly, a local folder is also a supported destination for the recording folder, say /Folder/MyRecordings/

That said, an EFS volume in AWS is a type of "virtual disk" that can be attached to multiple machines simultaneously, unlike the EBS volumes that can be attached only to one virtual machine and are usually intended for the boot drive/OS.

To the Linux OS, the media is not relevant because the EFS drive is formatted as an ext4 drive and mounted/mapped as a local folder, for example /home/Drive1

Since the EFS would be equally mapped to the primary and secondary servers as a local drive with the exact same path "/home/Drive1" it'd be exactly as using a SFTP server or Google bucket, as a single location for the recording files.

In theory it would still fall under the supported configurations. It'd be like in windows hooking up an external USB drive and pointing the recordings to it.

In another post about backups, someone suggested to handle the call recording backups separate from 3CX. It makes sense if i'll have 500GB in call recordings. That kind of a 3CX backup would take forever to complete, may hang along the way (been there), and it would take several hours to transfer out of AWS not to mention the internet data transfer costs they'll charge you.

Instead I can do incremental file system backups with rsync and crontab for example.

 

[Evolute IT]

In another post about backups, someone suggested to handle the call recording backups separate from 3CX. It makes sense if i'll have 500GB in call recordings. That kind of a 3CX backup would take forever to complete, may hang along the way (been there), and it would take several hours to transfer out of AWS not to mention the internet data transfer costs they'll charge you.

Instead I can do incremental file system backups with rsync and crontab for example.

Don't ever use rsync to try to sync 3CX instances. If your recordings are stored as you mentioned above, the backup itself shouldn't be too big as it will only contain text files, logs, voicemails and few other things (depending on what you chose to backup.) Those backups are meant to be used for this.

Using rsync or any other method of "synchronization" would result in errors as the database needs the infos on those files.

The passive server shall be configured to restore like 1h after the active backed up, so that in case of failover, you have the latest running.

The failover docs guide will help you set this up.

Thank you very much for all the help and the info, I really appreciate it. I'm using AWS free tear right now for a PoC, basically to understand how 3cx behaves under different conditions. I got the 1 year free license.

You mentioned SFTP or Google bucket. If i read the documentation correctly, a local folder is also a supported destination for the recording folder, say /Folder/MyRecordings/

That said, an EFS volume in AWS is a type of "virtual disk" that can be attached to multiple machines simultaneously, unlike the EBS volumes that can be attached only to one virtual machine and are usually intended for the boot drive/OS.

To the Linux OS, the media is not relevant because the EFS drive is formatted as an ext4 drive and mounted/mapped as a local folder, for example /home/Drive1

Since the EFS would be equally mapped to the primary and secondary servers as a local drive with the exact same path "/home/Drive1" it'd be exactly as using a SFTP server or Google bucket, as a single location for the recording files.

In theory it would still fall under the supported configurations. It'd be like in windows hooking up an external USB drive and pointing the recordings to it.

In theory, yes. It should work if it works as you mention. You just have to set the permissions correctly for the Linux user "phonesystem".

 

[svaira]

Hi, i'm Stefano...sorry for my english...and sorry if i interfere in this discussion...
I write here because i don't find element or guide explaining Failover...if neessary...
I also have many doubts about it.
I have a customer with 3 Office, with 3 different internet connection and each office has a backup of internet connection.
3CX is host in cloud 3CX (I haven't put the license yet)
Office A 50 operator (all users will use desktop app except 7 IP Phone Yealink)
Office B 20 operator (all users will use desktop app except 5 IP Phone Yealink)
Office C 5 operator (all users will use desktop App)
I create 2 sbc, 1 for office A and 1 for office B, Office C use only desktop app.
My idea was use or 1 license Enterprise or Pro...probably pro.
Now, the customer wants a failover.
I understand this, for failover needs 2 Host, 1 host in 3cx and another with another provider, correct?
The SLA of Digital Ocean?
Does it make sense to have failover? or is an HA sbc better? Bridge?
I'm not understanding anything anymore.
What is the most correct solution?
and in any case I don't find exhaustive guides that explain well
Can you help me?

 

[ChrisC_3CX]

Hello @svaira

3CX is host in cloud 3CX (I haven't put the license yet)

If the instances are hosted by 3CX then I'm afraid that the 3CX Failover functionality is not available.

I understand this, for failover needs 2 Host, 1 host in 3cx and another with another provider, correct?

Having 3CX in failover mode indeed means you'll need two hosts. They do not necessarily need to be on different providers but they must both either be local or remote(cloud). That said, you cannot use a 3CX Hosted instance for neither of the two hosts.

Does it make sense to have failover? or is an HA sbc better? Bridge?

3CX Failover, SBC HA and 3CX Bridge are three different things so it depends on what you're trying to achieve.

Very briefly:

3CX Failover is two 3CX PBX instances running at the same time, one being the Active server and the other being the Passive but both using the same License Key (so only one key required). The only one actually operating as the PBX is the Active server while the Passive server continuously monitors the Active server. In the event that the Active server fails (loses connectivity, has services drop, becomes unresponsive, etc) the Passive server will take it's place by automatically becoming the Active server. Note: If you're using a 3CX FQDN the DNS records for it will automatically be updated in case the public IP is a different one.

SBC HA: This is a configuration involving two SBC's running at the same time but only using a single local IP(virtual) to reach them. Again, there are two SBC's one acting as the passive and the other as the active and they switch upon failure of the active SBC. This will not replace the 3CX Failover as the 3CX Failover is for the PBX whereas the SBC HA is for a single SBC instance service a specific remote site. This will essentially eliminate a single point of failure for that specific remote site only in regards to the 3CX SBC.

3CX Bridge: The 3CX Bridge involves two or more 3CX PBX instances(Hosts). It is not to be confused with the 3CX Failover but rather, is most often used to allow communication between two or more 3CX PBX without having to use a SIP Trunk to accomplish this. That said, you can perform extension to extension calls from one PBX to the other, call out via a SIP Trunk that is actually on a different PBX and even see presence information(profile status, call status, etc) of extensions that are on a different PBX than yours, etc.

What is the most correct solution?

As previously mentioned, these three configurations are vastly different so which one you will need is completely up to you and your requirements. As they have different purposes you can of course use a combination of all three again depending on your requirements.

All relevant documentation of the mentioned functionalities can be found here:
3CX Failover: https://www.3cx.com/docs/failover/
3CX SBC HA: https://www.3cx.com/blog/releases/sbc-high-availability-clustering/
3CX Bridge: https://www.3cx.com/docs/manual/connecting-pbx-bridges/

For bridges there should be more information available in our 3CX Academy too.

Hope I've helped clear this up a bit.

 

[svaira]

Hello @ChrisC_3CX
Thanks very much for your reply.

If the instances are hosted by 3CX then I'm afraid that the 3CX Failover functionality is not available.

Why is not available? because they think about moving the machine from one node to another?
ok...now for the customer i've create a test enviroment, without license.
Recap...failover is possible cloud to cloud or onpremise to onpremise, but not possible with if the instance are hosting in 3cx, correct?

Having 3CX in failover mode indeed means you'll need two hosts. They do not necessarily need to be on different providers but they must both either be local or remote(cloud). That said, you cannot use a 3CX Hosted instance for neither of the two hosts

ok...i understand, because i can create 2 instance in the same cloud but with different location.
For the guide...thanks but it's my opinion...are very poor

 

[ChrisC_3CX]

Why is not available? because they think about moving the machine from one node to another?
ok...now for the customer i've create a test enviroment, without license.

This is just the case for a Hosted by 3CX Instance, one of the reasons being a security measure as the failover functionality allows you to run scripts on the machine .

Recap...failover is possible cloud to cloud or onpremise to onpremise, but not possible with if the instance are hosting in 3cx, correct?

That is correct. When we say cloud we essentially mean remote, so even if the instances are not hosted with a hosting provider, and are simply in two different remote offices then using the 3CX Failover configuration would still be possible.