Solved IVR audio stops working

[benknowles30]

I am running 3cx 15.5.15502.6. We are using an IVR Menu system which works fine but will stop working for a few times each work day. The calls are processed through our SIP Trunk and even register in the Call Log, but the caller will not hear any audio and any input like extensions will not work. I have to restart the Windows machine in order for everything to come back up and work. I have tried moving the phone system over to new hardware thinking it might be a hardware issue but the symptoms are the same. I test calling in every once in a while and have tried restarting the 3cx services one by one to see if it is a specific one but no one seems to clear up the issue. The only way to get it back is to restart the machine. After the audio stops going to the outside caller we are still able to make calls inside and even call out. Has anyone run into this issue?

 

[iversion-Support]

Hi Benknowles,
What operating system are you running the 3CX on, and the hardware specs please. Are you able to share the 3CX log? can you also check in the Windows event viewer any error entry register at the time of issue?

 

[benknowles30]

It is running on Windows Server 2016 Standard. Its a VM with 4 core, 8GB Memory, and 250GB storage space. The only errors I can see in the event viewer are some PostgreSQL errors when the server starts up, but nothing before or after the issue start happening.

I zipped up the Logs and have them shared here

 

[iversion-Support]

Hi benknowles30

Thank you for sharing the log with me, first of all I want you to know this issue is not cause by 3CX but the infrastructure its running on. You said its a virtual machine and you have shared virtual machine specs with me. Can you please share the actual host specs with me:

How much Ram, CPU type total number cores also the most important one, what type of storage are you using. Is it SSD, SAN, standard computer disk mechanical, server grade hardware RAID with 10k or 15K RPM disks.

What is important to understand is too often we virtualise without considering the underneath hardware capacity. For example if the actual physical box have standard HDD 7200 RPM then the through put of that disk is not enough to run two or more VM in production specially if the VM is running structure data like SQL.

Even if you have good SSD storage you need to make sure you thick provision the disk instead of thin if its running any type of database.

If you think hardware is good, please make sure when you create virtual machine you assign single socket and 4 cores not 4 socket one core each something you may over looked in VMware system.

If the storage and hardware is good, create VM disk with thick provisioning, backup your 3CX, install 3CX again using the backup and restore.

Let me know if this help, happy to investigate further.

Looking at the log i think your database is crashing and that is why the CPU , see system log below.

019/04/11 15:55:29.646|3452|0011|Erro|CHR: Worker thread caught exception: System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
at Npgsql.NpgsqlTransaction.CheckCompleted()
at Npgsql.NpgsqlTransaction.<Rollback>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlTransaction.Rollback()
at CallHistoryReceiver.DBAccess.CancelTransaction()
at CallHistoryReceiver.JoinedCall.SaveToDB(CallFlowParser parser)
at CallHistoryReceiver.CallFlowParser.ProcessFlows()
at CallHistoryReceiver.CHReceiver.WorkerMethod()
2019/04/11 15:55:29.667|3452|0011|Excpt|CHR: Npgsql.PostgresException (0x80004005): 57P03: the database system is shutting down
at Npgsql.NpgsqlConnector.<DoReadMessage>d__157.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

 

[benknowles30]

Thanks for the replies.

The host is running a 16 core Xeon with 96GB ECC Memory, this is one of 4 servers on a cluster that runs a CEPH hard drive array that runs enterprise 10k drives and some SSD drives. The host I put the phone server on is only really running an interface server that isn't used that often but is using a virtual network system.

I switched it to this cluster from another virtual host that was running an 8 core Atom with 8GB Memory and 1 ssd drive. The virtual machine that was setup on that host was a 4 core cpu with 5GB Memory and 1TB of space, running Windows 10 Enterprise. I was wondering if there was some sort of hardware issue because it started having this issue on that host.

I created another virtual machine with similar specs that I was going to try the 3cx linux iso to see if it was some sort of windows interference. Would you suggest that route or to put the phone server on its own hardware? Or maybe the processor information I gave to the virtual machine is causing issues? I noticed those SQL errors in the logs but don't know enough about the 3cx log system to really know if that was normal false positives or not.

Again I really appreciate the help.

 

[benknowles30]

Ive moved the configurations over to a linux server with the same vm specs as the windows machine. Things seemed to be working better; I didn't have the same sql errors as with the windows server. But today at 10am the audio stopped going to the inbound callers and I had to reset the server to get it back. I have zipped up the support information and uploaded it here if anyone can lend a hand.

I am putting together a standalone machine with the linux installation on it to move this over too soon to try and remove any possible hardware issues. Thanks for the help

 

[iversion-Support]

I don't thing there is any issues with your hardware specs base on the information you provided. If it is working on Windows may i ask which version of Linux you were using. Is it Debian or different flavor.

For you Windows setup, I can see in the log Call manager / Sip Server service was stopped at 10:01 see attached from you log.

May I ask what version of the Windows OS you are using. I am not sure what is causing the service to stop but I found another thread for the same issue with Windows 7 you may want to have a look at it.

https://www.3cx.com/community/threads/3cx-11-in-vm-service-got-signal-to-stop-on-reboot.33433/

https://www.3cx.com/community/threads/3cx-service-stop-by-itself.37525/

Can you please disable Windows Updates, also check backup etc. does it fail at the same time.

As a work around on Windows you could go in services and set each 3CX service to recover automatically in the services area. select the service, click on recovery option, set the service to restart on first and second failure see attached image.

 

[benknowles30]

Ive switched it from Windows over to the debian iso that is given by 3cx.

It runs great until the Phone Tree stops sending the audio. We have an additional line through the same trunk that I got working today. When the phone tree stops playing for incoming calls I can call the secondary number that is just set to a ring group and it still works fine. I have set the primary number to the same ring group as the second to see if it crashes still. Might have something to do with the Digital Reception system we have setup. The IVR log shows up to around 80 calls before it stops working. There is no lack of memory or hard drive space at this time.

When it was running on Windows I tried restarting each of the 3cx services to see if there was on that happened to stop. When the phone tree audio stopped for incoming calls all the 3cx services were still running. The only one that isn't is the hotel service but that wasn't ever on.

Edit: The same problem happened three times during business hours today so I installed the same 3cx linux iso onto its own hardware so its no longer a VM. It crashed about an hour after it was up and running. Thats when I turned off the Digital Assistant and went back to the Ring Group.

 

[iversion-Support]

Not, sure mate. Don't have these sort of issue on our installs. For 3CX services you can check them in the 3CX management portal. If they are green then they are working. How did you record the audio for 3CX?

Is it support format, check this out...https://www.3cx.com/docs/converting-wav-file/

But other then that I need to have a look at you call flow to go further.

 

[benknowles30]

It was a wav file and I followed that tutorial exactly with Audacity to convert it to the 16bit mono 8000Mhz format. I don't really think its the problem cause it was working for 2 months with no issues. Just at a loss and have started undoing anything I can think of to find the issue.

The company that put the system in place is non responsive to get any actual 3cx support so I am working on becoming a partner so i can take over the account and get some support from the company. Wish I could just purchase the support...

Thanks for the great insight, will keep this updated with any changes.

 

[iversion-Support]

You can purchase a support from 3CX as customer as well, you can also find another partner who can help you out.

Login to customer portal and buy support credit.

 

[benknowles30]

It appears to have been a NAT randomizing issue. The firewall test had never been run by the company that installed the phone system so it was always red, but it worked. Didn't think much of it.

After running the firewall test and getting it to pass all the way. Only required extending the port range to the version 16 set and shutting off random NAT. Haven't had a similar issue since.

Thanks for all the help!