Debian 15.5.5 Activity Log Query - CSTA & ParseException events

[bdarcyevans]

The system is running fine, but I've spotted a couple of events in the logs that I'm uncomfortable with. Would appreciate feedback from others if these are considered normal on a Debian installation or not.

1. When Logging level is set to Medium, "Trying to establish CSTA session to <sip:[extension]@[internal 3CX IP]:5062>" is logged against some extensions every minute. The phones are mixture of Yealink T4X series and are all up to date as per the firmware versions deployed through the 3CX console. All phones have been rebooted since the 15.5.5 upgrade. I have tried a "Reprovision" on these phones, but not a factory reset. Other than making sure they're running a firmware version that has CSTA support, are there any other steps required? AFAIK the phones that aren't throwing up this warning have been rebooted since the upgrade, but that's it.

2. Exception: ParseException ../../rutil/ParseBuffer.hxx:230, Parse failed unexpected eof in context: is being written to the activity log (visible in Low Logging level) sometimes when unsuccessful calls are being made - e.g. those that subsequently return a 404 Not Found or 502 Bad Gateway error. I have never seen this before on previous installations.

Your advice please.

Brendan

 

[cobaltit]

I've not noticed the first error (although honestly I haven't looked). I have seen the second error and 3CX support was less than helpful. We opened a ticket because we would see this error along with a sudden refusal of calls to go out via the SIP provider. 3CX support was focused on why the provider was rejecting the calls, although my thought was it was the parse error that was causing the outbound call failures. We had issues for almost 3 weeks after the v15 upgrade but then it just stopped with no changes. Only saw this on one customer but definitely not a warm fuzzy feeling. It did lead me to believe that 3CX is based on the reSIProcate project though.

 

[complex1]

Hi,

"Trying to establish CSTA session to <sip:[extension]@[internal 3CX IP]:5062>"
This mean the phone does not support or understand the uaCSTA feature.

 

[bdarcyevans]

Installed the recent 15.5 Update 1 (SP1) but still getting 'ParseException ../../rutil/ParseBuffer.hxx:230, Parse failed unexpected eof in context' in the Activity Log. For example;

08/11/2017 11:55:18 AM - Exception: ParseException ../../rutil/ParseBuffer.hxx:230, Parse failed unexpected eof in context:
<dialled number>
^
@ ../../rutil/ParseBuffer.hxx:230

Has anyone else seen this, or is anyone else experiencing this? I have contacted our 3CX partner who in turn contacted 3CX support on our behalf but neither of us have received a response.

 

[YiannisH_3CX]

Do you have a ticket number so i can take a look at the issue?

 

[bdarcyevans]

Thanks for reaching out @YiannisH_3CX. I'm waiting for our partner to provide me with the ticket number. In the interim maybe you could PM me and I can reply with our partners name and you could look it up that way?

 

[cobaltit]

@YiannisH_3CX

We opened a ticket as we had this issue with v15 SP5 I believe. Support put the blame on the provider rather than answer the pretty simple question as to why we were getting parse errors.There shouldn't be anything a provider could do that would cause a parse error unless it is a bug in the software. Ticket #145062 for reference.

 

[nb]

You are not entirely correct.
A Parse exception means that there is a sip packet sent to us that is formatted wrongly so our sip engine cannot parse the headers and the contents of the packet. Of course a provider has an intimate connection with a problem like this. Whoever is involved in a sip transaction that generates a parse exception is at fault. So it can be 3CX not understanding something standard that it should understand (highly unlikely because there are over 100 providers that do not show sign of this parse errors and some of them are mega telco's) or the provider doing something illegal. When phones do stuff that generate this exception we quickly tend to go check with the manufacturers to see why we have an exception. Same has to be done here.

In your case Cbeyond is the provider being used. In the 15.5 repository I cannot find a CBeyond template - which means that this provider is no longer supported. In this case, you have to do the legwork to communicate with the provider and ask them why 3CX is giving a parse exception.

Also Lets say you get this parse exception - What high level problems do you have? If you have no functionality issues treat it as a warning and continue. If these exceptions bother you, the only way is to use a supported provider OR escalate this to the provider and they will come to us if we are at fault.

 

[PhatPanda]

I am on Debian 15.5.5 as well and get many of the same parse errors, using fully supported phones and fully support SIP trunk from VOIP Innovations:

08/21/2017 9:17:10 AM - Exception: ParseException ../../rutil/ParseBuffer.hxx:230, Parse failed unexpected eof in context: 41 ^ @ ../../rutil/ParseBuffer.hxx:230

 

[nb]

OK then this we will check. If it is supported then we have the direct contacts to core NOC teams of the provider.
But we will need your support ticket and your full verbose logs, the time when you see the parse exception and copy paste of the log from the Management console.

Also you need to explain if you experience any side effects in the PBX.
Do you notice anything?

 

[cobaltit]

@nickybrg

Appreciate the detailed response. In debug mode I would expect to see the parse exception. In production software I wouldn't expect to see that and I've never seen 3CX barf like that before. If it really is a malformed packet then it almost sounds like a DoS vulernability. As it stands, this customer has been using Cbeyond (now Birch) for over 3 years now and this issue didn't come up until we upgraded to V15. And this issue persisted for a few weeks and then went away and it hasn't happened again. I have several customers on Cbeyond/Birch with no issues. They used to be supported provider back around v11/v12.

I also saw this issue with another new customer that is testing out 3CX. I have to go back and check my notes but I want to say it was Flowroute who is a supported provider. And apparently PhatPanda is seeing it with a 3rd provider so it seems to point to a larger issue. Kinda reminds me of the t38 issue back in v15 SP-something where you guys made a change (undocumented in the change logs I believe) that was standards compliant but caused issues with multiple providers, certified or otherwise.

Anyways I documented it in the ticket but our issue was the system would run anywhere from a few hours to a few days and then calls would fail in both directions and we'd see the parse error. A reboot of the system would resolve it and then it would happen again. This happened after an initial upgrade from v12.5 on Windows to v15 on Debian (in a VM). Thinking it was related to some VSS issues when we'd backup the VM we moved it to Debian on bare metal and the issue followed. A few days after we opened the ticket the issue stopped and hasn't come back since.

 

[nb]

Why you should not see exceptions in production? Why should they be hidden?
I think these are verbose material and should be displayed so you raise the concern to the developer.
I agree that maybe if the log is set to low, then you should not see them. Can you confirm if you see this in low or normal logging?

It cannot be a dos attack coming from Cbeyond - well it can, but it is highly unlikely because the dos material will have to pass from the provider's hops and it will be blocked/ rejected. V11/12 is a long time ago. Had much less features and much less checks. If v15 generates this, then it simply has to be checked.

I am not surprised there are these exceptions with an unsupported provider. BUT if you say that the result of this is so drastic (as in calls stop coming from all directions) then we really would like to see this.

PM me with details and I will log in to your system and check. If you have logs and verbose material in the ticket system we can work to check what is happening.

But in the meantime, why not speed things up and forward this to the provider? this should be something simple to do. We just need to have the SIP Packet that was sent when the exception was raised. Then I can look at it and I can see why the sip server is not parsing this. After a restart this stops - Well a restart resets the data container so then of course the exception or set of unhandled exceptions are thrown away or the malformed packet did not come after the restart.

We need the sip packet that generates this. You can help us because you are a client that works with a provider that causes 3CX to be unable to parse a packet. If you can trace that format, we will dig.

 

[cobaltit]

@nickybrg To elaborate, I would expect to see a more polished error in production, something along the lines of your explanation. So if I saw an error that said something like 'malformed SIP packet' then I would have a better idea of where to troubleshoot vs the parse buffer error.

And regarding the DoS possibility I didn't mean from Cbeyond. This particular customer has the SIP port open to the world because we have remote extensions so it could be a DoS attack from anywhere.

I appreciate the offer to help but this stopped shortly after the ticket and hasn't resurfaced. If I do come across it again I will definitely get logs and PCAPs and let you know. Your help/explanation here was definitely more helpful than the initial ticket response.

 

[nb]

Yes I understand what you mean - but you cannot expect polished errors in all cases. Thats a DEEP ERROR - deep in the core.. A parsing error in the sip engine. Maybe it is a malformed packet. I dont know yet. (Im talking only from experience)
Thats for us to check in debug and to send to the telco.

If you find the packet that causes the problem let me know.

Then we can talk further. If you see the contact not from CBeyond, then yes there is something that is sending traffic - but without authentication there is not much this something can do.