Huge logs

Discussion in '3CX Phone System - General' started by lucasasdelli, Jan 21, 2008.

Thread Status:
Not open for further replies.
  1. lucasasdelli

    Joined:
    Oct 28, 2007
    Messages:
    7
    Likes Received:
    0
    Hi all,

    since version 5.0.3790, I experience some system freezes due to huge log creation: the last night was something like 40GB log files.

    Here the message, repeated near infinite times:
    .\CM_Stack.cpp(163)|Critical0|SipStack|CallMgr::Stack::thread:[CM100002]: *!* Exception detected: bad allocation *!*<br>

    It seems that shutting down 3Cx' Apache webserver stops the issue.

    Cheers
    Luca
     
  2. josvic

    Joined:
    Apr 24, 2007
    Messages:
    51
    Likes Received:
    0
    I have the same issue... any solution?

    The unique way to stop then are to reiniciate the entire server..., please... any help, is that a bug in the program?

    Please... help....
     
  3. Alan van der Vyver

    Joined:
    Nov 15, 2007
    Messages:
    15
    Likes Received:
    0
    I have had the same problem on a test system that was not actively used or monitored for a while, except that I can't tell what the problem log entries were, because they completely filled the disk drive, to the point where it was a struggle to log on, because there was not enough space to update my profile.

    It also resulted in the corruption of the system database and, since it was a test system, I did not have a backup, so the system is off at the moment.

    The obvious solution for me, assuming the log messages were legitimate, is to schedule scripts to delete old logs, but it would seem a better idea for 3CX to have a mechanism to automatically do this.
     
  4. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello,
    Today I had the same problem. I have an 3CX running since several months ago, with no problems.
    Today, the partition where there is the 3CX was full, and looking for the problem, I detected that at
    C:/Program Files/3CX PhoneSystem/Data/logs/Backup/20080508
    was full of files
    3CXPhoneSystem.HHMMSS

    And these files are full of the same message:
    08:07:18.626|.\CM_Stack.cpp(163)|Critical0|SipStack|CallMgr::Stack::thread:[CM100002]: *!* Exception detected: bad allocation *!*<br>

    I stopped and started the service, but the logs kept appearing.

    Then, in the Web interface, in General Settings, I saw that the level of log was for debugging purposes, and I changed it to low level, so seems that this log is not generated again.

    I have the last version of the 3CX (5.1.41280), and I want to know if someone knows the reason of this log, and if I can do something to repair the problem.

    I have no VoIP providers. We use Linksys 3102 as Gateways, and each gateway has an analogic phone and a IP phone (3108 3Com).

    Thanks and best regards.
     
  5. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello,
    I detected in another PC with the 3CX the same problem at the same date, so it seems very weird.
    In the second PC, there was the debugging level of logs, too.
    I made the same steps, to remove the log files and change the debug level to low. Nevertheless, the log is no writing now, but the problem still ocurring.

    Thanks and best regards.
     
  6. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    I suspect you have installed some new hardware recently, or, probably had changed some settings. Exceptions in stack usually happens when there's badly formed SIP messages, for example with unreasonable long Contents-Length: 234234324234, or something like this. I would ask you to capture a SIP messages at one of the 3CX boxes at the moment when exceptions happens, if possible. Wireshark or Ethereal sniffers is just great for this task. Than I can try to find what is broken and make a fix in our SIP stack to eliminate exceptions.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    BTW, good place to look: check display names of your devices. They shouldn't contain any "funny" symbols. Especially - star symbol '*'.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Thanks for the answer.
    I don´t think any value was changed recently, the 3CX is working for several months, and until last week, we had no problems with it.

    I am capturing the packets with Wireshark, but at this moment there are no huge logs, and I can´t force them, because I don´t know why they produce themselves.

    When I get the huge logs, I will send the Wireshark capture.

    About device names, I don´t know if you refer gateway names, extension names, etc... I only use letters, numbers, some spaces and _. And these name are the same since several months.

    Thanks and best regards,
     
  9. polyphemos

    Joined:
    Feb 4, 2008
    Messages:
    16
    Likes Received:
    0
    I can also report the very same error at practically the same time as rduran, and possibly I have also located what triggered it.

    My system (5.1.4128.0 on an XP machine) started generating the sipstack error messages at 00.15.50 on 8 May (Swedish time = GMT+1). It kept going until it had filled up its partition with some 12 GB a few hours later.

    But before this, it had spent the previous two days trying to speak to the 3CX STUN server at 64.69.76.23, generating a timeout error every three seconds. I don't know if the server was down or if there was some other problem, but it's clear from the logs that my system did not get a proper response anytime from 00.32.06 on 6 May, until it stopped trying and went into the sipstack error loop.

    Would be interesting to know if rduran can see the same pattern on his two machines.

    The logs look like this:

    [6 May]

    [last normal STUN request]
    00:32:06.513|.\StunClient.cpp(210)|Log2|STUN|StunClient::eek:nInitTests:[CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 64.69.76.23:3478 over Transport 192.168.0.2:5060<br>

    [timeout errors start]
    00:52:03.783|.\StunClient.cpp(210)|Log2|STUN|StunClient::eek:nInitTests:[CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 64.69.76.23:3478 over Transport 192.168.0.2:5060<br>
    00:52:06.837|.\StunClient.cpp(252)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.0.2:5060<br>
    00:52:09.892|.\StunClient.cpp(252)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.0.2:5060<br>
    00:52:12.963|.\StunClient.cpp(252)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport:


    [continues every three seconds until]

    [8 May]

    00:15:50.207|.\StunClient.cpp(252)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.0.2:5060<br>
    00:15:50.441|.\CM_Stack.cpp(163)|Critical0|SipStack|CallMgr::Stack::thread:[CM100002]: *!* Exception detected: bad allocation *!*<br>
    00:15:50.441|.\CM_Stack.cpp(163)|Critical0|SipStack|CallMgr::Stack::thread:[CM100002]: *!* Exception detected: bad allocation *!*<br>
    00:15:50.441|.\CM_Stack.cpp(163)|Critical0|SipStack|CallMgr::Stack::thread:[CM100002]: *!* Exception detected: bad allocation *!*<br>
     
  10. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    Thank you for input, I will check for memory leaks in case of STUN request failed. Meanwhile I would suggest you to upgrade to latest V5.1 build which is 4510.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello,
    My problem was the same, with the STUN Server error. Sorry because I didn´t mencioned it, but I didn´t relation the two issues.
    I can show the logs in the second server,
    The problems with the STUN server started on 6th,

    13:30:42.640|.\StunClient.cpp(210)|Log2|STUN|StunClient::eek:nInitTests:[CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 64.69.76.23:3478 over Transport 192.168.99.5:5060<br>
    13:30:42.843|.\StunClient.cpp(115)|Log2|STUN|StunClient::process:[CM506003]: Resolved SIP external IP:port has changed to (213.60.130.224:4247) on Transport 192.168.99.5:5060<br>
    13:50:42.999|.\StunClient.cpp(210)|Log2|STUN|StunClient::eek:nInitTests:[CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 64.69.76.23:3478 over Transport 192.168.99.5:5060<br>
    13:50:43.203|.\StunClient.cpp(115)|Log2|STUN|StunClient::process:[CM506003]: Resolved SIP external IP:port has changed to (213.60.130.224:5517) on Transport 192.168.99.5:5060<br>


    Later, on same day, the logs changed to
    00:55:25.266|.\StunClient.cpp(210)|Log2|STUN|StunClient::eek:nInitTests:[CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 64.69.76.23:3478 over Transport 192.168.99.5:5060<br>
    00:55:28.325|.\StunClient.cpp(250)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.99.5:5060<br>
    00:55:31.385|.\StunClient.cpp(250)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.99.5:5060<br>
    00:55:34.445|.\StunClient.cpp(250)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.99.5:5060<br>
    00:55:37.504|.\StunClient.cpp(250)|Log2|STUN|StunClient::eek:nTestTout:[CM506004]: STUN request to STUN server 64.69.76.23 has timed out; used Transport: 192.168.99.5:5060<br>


    And finally, on 8th in the morning, started the logs with the exception until the hard disk was full, and I rebooted the 3CX and changed the logs lo low level.

    Yesterday, I tried to capture SIP packets in the error, but I wasn´t able to reproduce the issue.
    I hope we can have a solution to resolve this.

    Thanks
     
  12. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello,
    This week the problem happened again, so I had to reboot the 3CX.
    I don´t know if someone has detected the issue, and there is some solution.
    I still have the same 3CX version: 5.1.4128.0. I don´t know if it is recommended to upgrade to solution this.

    Thanks and best regards,
     
  13. Anonymous

    Anonymous Guest

    Rduran

    1) What firewall is being used here?
    2) Is this a dynamic IP?
     
  14. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello Keith,

    The 3CX Server is in a Windows 2003 Server in a subnet with static IP. The phones are in another subnet, but the router make connectivity at level 3.
    There are some firewall rules: the only traffic allowed between 3CX Server and IP phones are: 5060 UDP, 5061 UDP and 9000 - 9009 UDP. The system is working ok during months, and suddenly this issue happens, so I don´t think this have relation with ports and firewall rules.

    Thanks for your help
     
  15. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    My home router periodically stops to forward requests and I can not browse internet during that time. It can work OK a month or so, though. I don't think it's a bug of Firefox or IE.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Hello Archie,

    Sorry if I have no express myself correctly. I don´t use to speak English, so my message can seem angry. My problem is no very important, and the 3CX is working ok, so I have no complaint. I only want to know if someone know the source of this problem, to correct it.

    I think there is no problem with the firewall, but I can be wrong. If you think I have to check the firewall policies, and add some port to permit a better comunication there is no problem.

    Thanks for your help,
    Rduran
     
  17. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    No, I didn't take your message as an angry post. I'm just trying to tell that sometimes hardware also may fault. And it is much more difficult to diagnose such faults. As in example with my home router. I still don't know what happens when I lose access to internet. It just stops to work. Though I do not touch any forwarding or settings of router. After reboot it again works OK.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Ok.

    I don´t give this issue priority, because restart the system once each several months, is no problem. The problem is that I have the 3CX in the same partition that the operating system, so the huge amount of logs use all the partition. I think I will move the 3CX to another partition to avoid this issue.

    Thanks for your help,
    Rduran
     
  19. archie

    archie Well-Known Member
    3CX Support

    Joined:
    Aug 18, 2006
    Messages:
    1,299
    Likes Received:
    0
    You can limit the amount of logs 3CX will write and keep. But this requires manual configuring of 3CXPhoneSystem.ini files. For the beginning try to set logs verbosity to minimum. May be it will be enough for you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. rduran

    Joined:
    May 8, 2008
    Messages:
    9
    Likes Received:
    0
    Yes, the first time I had the log level in verbose level, but then I changed it to minimum, so it has the minimum level now.
    I opened the 3CXPhoneSystem.ini, and during the file, I find Severity=Log Error Critical

    Thanks,
     
Thread Status:
Not open for further replies.