Runaway 3CXVoiceBoxManager service

[paragondie]

When we first turned on the 3CX system the email notifications were working fine but after a few days the system locked up. Investigation showed that 3CXVoiceBoxManager.exe was using 100% of the processor and the trace log filled the hard drive. After shutting off the service we did some testing and found that there were two voicemails that would trigger this behavior. We could find nothing unusual about these voice mails. In fact at least one of them had been on the server for two days before we had the problem. We deleted the two voice mails and restarted the service. A few hours later the same thing happened. We currently have the service disabled. The trace.log file contains these lines at the point the system runs amuck:

09:02:54.125|0598|.\TasksProcessor.cpp(753):CTaskSendNotify::SendNotify(), Notification for the 'from 02 Oct 2007 12:00, caller 20004' message ('138' extension) has been sent.
09:02:54.125|0598|.\PhpWrapper.cpp(381)HPWrapper::Execute(), Process 0x344 (D:\Program Files\3CX PhoneSystem\Bin\..\Bin\Php\php-win.exe) created
09:02:54.125|0598|.\PhpWrapper.cpp(382)HPWrapper::Execute(), Writing 326 bytes of input data
09:02:54.125|0598|.\PhpWrapper.cpp(394)HPWrapper::Execute(), Input data written for 0x344 process
09:02:54.125|0598|.\PhpWrapper.cpp(406)HPWrapper::Execute(), Waiting for finishing of 0x344 process, timeout is 5000 ms
09:02:54.313|0598|.\PhpWrapper.cpp(412)HPWrapper::Execute(), Process 0x344 finished
09:02:54.313|0598|.\PhpWrapper.cpp(431)HPWrapper::Execute(), Waiting for data from process
09:02:54.375|0598|.\PhpWrapper.cpp(433)HPWrapper::Execute(), Data from process gotten
09:02:54.547|05a0|.\VoiceBoxScanner.cpp(235):CVoiceBoxScanner::WatchFolders(), Error! Exception detected
09:02:54.547|05a0|.\VBoxAgent.cpp(55):VBoxAgent::WatchingThread::Run(), Error of voicemail folders watching
09:02:54.547|05a0|.\VoiceBoxScanner.cpp(235):CVoiceBoxScanner::WatchFolders(), Error! Exception detected
09:02:54.547|05a0|.\VBoxAgent.cpp(55):VBoxAgent::WatchingThread::Run(), Error of voicemail folders watching
09:02:54.547|05a0|.\VoiceBoxScanner.cpp(235):CVoiceBoxScanner::WatchFolders(), Error! Exception detected
09:02:54.547|05a0|.\VBoxAgent.cpp(55):VBoxAgent::WatchingThread::Run(), Error of voicemail folders watching
09:02:54.547|05a0|.\VoiceBoxScanner.cpp(235):CVoiceBoxScanner::WatchFolders(), Error! Exception detected
09:02:54.547|05a0|.\VBoxAgent.cpp(55):VBoxAgent::WatchingThread::Run(), Error of voicemail folders watching

These lasts two lines repeat continually and very rapidly causing the log file to quickly consume gigabytes of space.

Has anyone else had this problem and has anyone found a solution?

 

[miraportuga]

Seems to be an software problem, reinstall ? :|

 

[paragondie]

Did that ...

Actually we just reinstalled the software on a new box running Virtual Iron for a virtual server, same result.

 

[Mirzab]

Would be nice if you could supply additional info regarding specs and OS of first system and current virtual. Was first install virtual also?

Since voicemail is cause and stored as media files I would speculate your Anti-Virus software may be the culprit. Exempt 3CX folder and outbound email for testing.

 

[paragondie]

First install: Dell Precision 530 (1.7GHz Xeon w/2GB RAM) running Windows XP Pro SP2
Second install: Same system/same operating system
Third install: Dell PowerEdge 4600 (2.4GHz Xeon w/2GB RAM) running Windows 2003 Server R2 Enterprise (trial) SP2
Fourth install: Virtual Iron VM (Dual 2.66GHz Xeon w/4GB RAM) running Windows 2003 Server R2 Enterprise (full) SP2

Installs one and two included anti virus. Installs three and four did not.

 

[LORD ORION]

What about sharing and security on the folder?

Did you install 3CX using admin priviledges?

If you are not logged into the account that you used to install 3CX, does the user have permission to acces the folder? Can enough concurrent users be connected when considering sharing?

This happens on other software when a temp file is created during recording and then finalaized and written to another folder.

 

[paragondie]

3CX was installed using the Administrator account. Permissions on those folders are set by the 3CX voice mail system as it creates them. No sharing is turned on for the system. The users only have access to their voice mail through the voice mail system.

As mentioned above, if we delete the voice mail that's there the system will work fine for a while but eventually fails again. We thought it might be the quantity of voice mail but we narrowed it down to a specific voice mail message that crashed it the first time. If we moved that file out of the voice mail folders the service would start back up fine. If we moved all other voice mail and put that one file back the service would jump immediately to 100% CPU as soon as it was started so it's not an issue with the quantity of voice mail or voice mail folders.

 

[LORD ORION]

Agreed,

But I have seen it more then once on other software recording to a temp file fail to finalize and write, connect concurrently and get stuck forever.

eg: SQL write BLOB or Virtual Machine timing problems in telephony apps, which is why I mentioned it. Typically it is sovled by allowing as many concurrent connections to the folder as you have potential simultaneous extensions in use, and or by simply allowing everyone write access to the folder in security to verify that is the problem. (especially true in MSS)

What is interesting is that the system gets gummed up when you copy the file back over.

I wonder what is so special about it.
eg: Not finalized? Corrupted?

 

[paragondie]

Have done some more testing. Turns out it's the number of folders that's the problem. It doesn't matter if there are messages in the folder or not. If there are more than 63 folders in the Extensions folder then the service begins generating errors and takes over whatever processor it's running on. The only solution at that point is to stop the service, reduce the number of folders to 63 or less and restart the service. Is there a limit to the number of voice mailboxes that 3CX supports? We could find no such limit in the documentation. We have nearly 100 at this point but we only run into this problem once the actual folders are created. As a work around we are running a script that deletes empty folders periodically but it's only a matter of time before there are more than 63 folders that actually have voice mail in them.

 

[RobLloyd]

That sounds like a bug. Good catch on that.

 

[Alexander]

Hi to All !

If somebody wants to test patch please send me ([email protected]) a message. I'll send the patch as a response.

Thanks
Alexander

 

[paragondie]

Is this patch included in v4 that has just been released or will we need the fix to load on top of v4?

 

[Alexander]

Hi !

This patch is included to version 4. But for now version 4 exists only as alpha. So if somebody wants to postpone installation of version 4, then he can use patch.

Thanks
Alexander

 

[paragondie]

Thank you Alexander. We are going to load alpha on a test box and do some testing.