Flipping Trunks

[dkirk-ads]

3CX v16 Update 2 running on Debian. The issue really started under Update 1 but since I upgraded to Update 2 last week the problem is really hitting us hard. Under Update 1 our secondary SIP Trunk would drop off and become Red/Unusable due to the generic event logged as:

Registration at SIP.US Secondary has failed. Destination (sip:[email protected]:5060) is not reachable, DNS error resolving FQDN, or service is not available.

If I hard-code the IP address or leave it FQDN formatted, the same message, so it's not a DNS issue. The service is always pingable and a tcpdump on the SIP.us provider side shows no packet activity on their end from my 3CX server.

After a few hours this same secondary service would suddenly reappear and function for several more hours until it failed again. The times appeared to be purely random, the secondary trunk would cylce on/off. The secondary would just randomly go down and then back up again a few times per day. Enter v16 Update 2.

Since the Update 2 upgrade the primary and secondary trunks flip between Active and Deactivated, again randomly. Last evening at 7:26pm I received this e-mail from the 3CX Server:

Trunk L:10001(SIP.US Secondary) has changed status to registered.

Followed immediately by this second e-mail:

Trunk L:10000(SIP.US Primary) has changed status to unregistered. This means that no more calls will pass via this trunk

The Primary was running but the Secondary suddenly tripped back active, triggering the Primary to then become disabled, immediately. I have removed/deleted the secondary trunk definition, rebooted, and entered it back in again but nothing changes. Again, the trunk provider never sees any traffic from my 3CX server to their secondary IP address when it is disabled and tries logging in every minute.

The 3CX firewall checker passes fully green and the firewall is a Sonicwall TZ500 with the latest firmware loaded.

For a week now I see these two trunks flipping back and forth and I've reached the limits of what else I can think to try, and the trunk provider has no options since I'm not even hitting their server when the trunk is down. When the primary trunk fails due to the secondary coming online that shows the networking and firewall must be fine, but there is something deep inside 3CX that isn't happy. The trunk provider is 3CX supported so I'm using their default templates. Just to be sure the Sonicwall firewall and Debian server were both rebooted.

Since the Update 2 I have tried removing the secondary Trunk and adding it back again, but each time it fails to register.

The IP BlackList is clear on the 3CX server and has never indicated being connected to the problem.

This morning when the active trunk was the Secondary and it was showing "Green" status, we were unable to place any outgoing calls. Restarting the SIP server services brought the primary back online and the secondary remained disabled.




Something I have not tried yet is do delete both primary and secondary trunks and reload them back in again. If I remove the primary will that also remove any dialing rules or other settings?

Any other suggestions out there? I tried to purchase a support ticket from the 3CX customer portal but all I get is a spinning set of circles, nothing ever displays allowing me to purchase a ticket.

 

[JohnS_3CX]

Hi

Can you use gw1.sip.us on both? Its not easy to tell, but in case gw2 went down or was inaccessible for whatever reason, and since gw1 seems stable then it might be worth to stay on the primary server as a test for a few days

 

[dkirk-ads]

The server gw2.sip.us isn't the problem, it is always ping-able as well as runs well when GW1 decides to go down. I would think having two gw1 connections, if even allowed, would cause confusion with both 3cx and the trunk provider.

 

[JohnS_3CX]

You have a couple of ways to "catch" the issue here, one way is to increase the logging level to verbose and when the trunk goes down look at the registrations around that timestamp.

Another way would be to keep tshark running in the background for packet capture, and when the issue occurs take a look at the capture to see whether the PBX got a response when it tried to refresh the registration.

This way you can see what the provider responded with during the event (or if you got a response at all) and take it from there

 

[dkirk-ads]

Went into Verbose logging for five minutes, long enough to ensure gw2 was tried. Which of the log files should have the information we're looking for? I pulled down all of the .log files and didn't see anything related to any gw2. connection.

 

[JohnS_3CX]

Ok at this stage I would suggest to download the 3CX Log Viewer and open your *.bldef file with it (found in the "Logs" folder of the Support Info package) https://www.3cx.com/docs/3cx-log-viewer/

We will first filter to find registration attempts where the PBX asks your provider to register, and then we will see whether we got a reply (and what the reply was in this case)

1. Find you trunk ID by going to the trunks in MC, and type 10000 or 10001. Mine here will be 10000, lets assume it is this one for the example below.

2. Now that we know the trunk ID, open the Bin Log Viewer and load your .bldef file

3. Lets filter for registration attempts now, by entering in the Filtering Options (SipMsg.Method(REGISTER) And DN(10000)) where 10000 is out trunk we wish to check of course

4. This will show us the attempts made to register the trunk we want to check, but we also want the replies from the provider to show up. Click on the result that appears in the Log Window and the Tags Window will show us some useful information, specifically the SipMsg.Call-ID

5. We empty the Filter formula text now that we know the Call-ID, and double click SipMsg.Call-ID to use it as our filter, for example you will see something like this:
SipMsg.Call-ID(0tP2QNCj8qQk6Q44FDakZA..)
Click apply to show the new results now with this ID as our filter

6. Now our Logs window finally shows the messages we are interested in: Registration and reply from provider



You will notice that I have two messages here:

a) the first one above with the arrow pointing to the corresponding entry as seen in the details window is Sent Registration for LC:10000 and contains the REGISTER method

b) The second one Updating Device Dev is the reply from my provider, which shows the 200 OK when I select it and look in the details window again




So this is what you would expect to see more or less for your successful attempts. For the unsuccessful ones you will either not see a reply at all (causing failure) or maybe you will see a reply from the provider with an error message informing you why they rejected your attempt

 

[dkirk-ads]

What a fantastic introduction to the log viewer, I was totally unaware of it, thank you. To answer your question, I see the "Sent" messages but I never get a reply to any of them.

A small portion of what was sent, with no reply, and with address hard-coded this time:

16/07/2019 14:29:31:699 | 8 | Sent registration forLc:10001(@SIP.US Secondary[<sip:[email protected]:5060>]) :ClientReg Send Req REGISTER from 0.0.0.0:0 tid=f5a3bc77ff859 Call-ID=Kjiz--J6yGzLI7FPIu .:
REGISTER sip:74.81.71.18:5060 SIP/2.0
Via: SIP/2.0/ ;branch=z9hG -524287-1---f5a3bc77ff 346;rport
Max-Forwards: 70

 

[JohnS_3CX]

Ok that means the PBX rightfully informed you that the trunk was down. No reply reached the PBX in this case.

I think the next step would also be to capture some traffic on the firewall if you have the capability to do so. It's important to see if the provider sent any traffic to your public IP at least on the edge device, which would normally be your firewall, just to ensure that the firewall didn't drop the packet and eliminate this possibility.

 

[dkirk-ads]

SonicWall TZ500. The provider ran tcpdump and never saw a single packet from me on the gw2. network. I'll see about capturing the firewall traffic. Smells like a firewall rule that may not be just right.

 

[JohnS_3CX]

Might be, the capture should reveal some useful information