Solved SBC version number changes back and forth

[SteveITS]

I don't understand this but figured I should post just to see if anyone else has seen this. Today I was looking at a client's system and it showed the SBC had an update available. I thought that was odd and double checked the ticket from a few weeks ago where I wrote that I upgraded it. It was showing in the trunk page as (IIRC) 16.0.x with 16.2.24 available (shows Update mark, check box, click Update to see the version). Figuring it didn't actually update for some reason back then, I reopened my ticket.

This evening I log in to do the update and it's showing 16.2.24. The Update marker is not shown and the Update button is disabled. The SBC hasn't restarted because it shows a registration timestamp of 2 days ago and we didn't get any alerts for it today. I even checked all our other servers and they all show 16.2, so I wasn't looking at the wrong one.

 

[SteveITS]

It is still doing it:

2 min later (screen cap, create this post, click to try to copy version which opened the SBC trunk, go back to trunk page):

PS - hooray, I'm not crazy.

 

[cobaltit]

Any chance this is a failover setup and the 2nd SBC didn't upgrade? Otherwise I'd just blow away the config/SBC software and reinstall.

 

[SteveITS]

No it's not a failover. Only one Pi. It's been fine since installed in March.

 

[JohnS_3CX]

Hi Steve,

You want to log on to your Pi and check /var/log/dpkg.log to see what's going on.

If you don't actually find history of any version changes, the @cobaltit may be on to something, so better check what's hooked up to the network. Perhaps someone installed something you are not aware of?

 

[SteveITS]

/var/log/dpkg.log is size 0.

/var/log/dpkg.log.1:
2020-11-13 17:02:21 startup archives unpack
2020-11-13 17:02:21 upgrade 3cxsbc:armhf 16.0.390 16.2.24
2020-11-13 17:02:21 status half-configured 3cxsbc:armhf 16.0.390
2020-11-13 17:02:31 status unpacked 3cxsbc:armhf 16.0.390
2020-11-13 17:02:31 status half-installed 3cxsbc:armhf 16.0.390
2020-11-13 17:02:31 status half-installed 3cxsbc:armhf 16.0.390
2020-11-13 17:02:31 status unpacked 3cxsbc:armhf 16.2.24
2020-11-13 17:02:31 status unpacked 3cxsbc:armhf 16.2.24
2020-11-13 17:02:32 startup packages configure
2020-11-13 17:02:32 configure 3cxsbc:armhf 16.2.24 <none>
2020-11-13 17:02:32 status unpacked 3cxsbc:armhf 16.2.24
2020-11-13 17:02:32 status unpacked 3cxsbc:armhf 16.2.24
2020-11-13 17:02:32 status half-configured 3cxsbc:armhf 16.2.24
2020-11-13 17:02:33 status installed 3cxsbc:armhf 16.2.24

dpkg.log.2.gz is from Jan. 30.

 

[JohnS_3CX]

This is indeed a bit strange, but do your recent logs show any activity? (ie. from the last 5-6 days)

Also, can you confirm you haven't installed any 3rd party monitoring tools, maintenance tools, remote management tools or anything that may be trying to auto-update the installation in your Pi?

 

[SteveITS]

do your recent logs show any activity

Hi John, the most notable event on the 3CX server is " 3CX logs and temporary files were deleted to conserve disk space" so nothing of use. I'm not seeing anything in the activity log..anything specific to try as a search term?

On the SBC, /var/log/messages has only "rsyslogd was HUPed" for 3 days.
3cxsbc.log ends a week ago with connection issues (believe this was the day the local ISP had problems):

CRIT | 20201202-003817.968 | 3CX | SBC | 1996376592 | RPiTunnel.cpp:390 | Stopping: global connection timeout
CRIT | 20201202-003817.988 | 3CX | SBC | 1996376592 | RPiTunnel.cpp:390 | Stopping: global connection timeout
CRIT | 20201202-003818.009 | 3CX | SBC | 1996376592 | RPiTunnel.cpp:390 | Stopping: global connection timeout
CRIT | 20201202-003818.075 | 3CX | SBC | 1996376592 | RPiTunnel.cpp:385 | ** Main console thread exits.

CRIT | 20201202-003826.647 | 3CX | SBC | 1995688464 | Log.cpp:158 | ====================== 3CX SmartSBC 16.2.24 @ ch-3cx-sbc ======================
CRIT | 20201202-003826.648 | 3CX | SBC | 1995688464 | Log.cpp:159 | Log started: type = file, level = ERR, size = 104857600, resip = ERR, remote = -1
ERR | 20201202-003906.691 | 3CX | SBC | 1995688464 | Tools.cpp:61 | getaddrinfo error: Temporary failure in name resolution
ERR | 20201202-004116.112 | 3CX | SBC | 1995688464 | Tools.cpp:95 | Failed to connect to 8.8.8.8:53, error 110
ERR | 20201202-004329.232 | 3CX | SBC | 1995688464 | Tools.cpp:95 | Failed to connect to 8.8.4.4:53, error 110
CRIT | 20201202-004329.233 | 3CX | SBC | 1995688464 | RPiTunnel.cpp:83 | Failed to build bridge by config [Bridge/123456]. Error: Invalid bridge's configuration!
  ** Failed to select network interface based on tunnel address and port
CRIT | 20201202-004337.944 | 3CX | SBC | 1996229136 | Log.cpp:158 | ====================== 3CX SmartSBC 16.2.24 @ ch-3cx-sbc ======================
CRIT | 20201202-004337.944 | 3CX | SBC | 1996229136 | Log.cpp:159 | Log started: type = file, level = ERR, size = 104857600, resip = ERR, remote = -1
ERR | 20201202-004417.988 | 3CX | SBC | 1996229136 | Tools.cpp:61 | getaddrinfo error: Temporary failure in name resolution
ERR | 20201202-004628.432 | 3CX | SBC | 1996229136 | Tools.cpp:95 | Failed to connect to 8.8.8.8:53, error 110
ERR | 20201202-004841.552 | 3CX | SBC | 1996229136 | Tools.cpp:95 | Failed to connect to 8.8.4.4:53, error 110
CRIT | 20201202-004841.553 | 3CX | SBC | 1996229136 | RPiTunnel.cpp:83 | Failed to build bridge by config [Bridge/123456]. Error: Invalid bridge's configuration!
  ** Failed to select network interface based on tunnel address and port
CRIT | 20201202-004852.241 | 3CX | SBC | 1995921936 | Log.cpp:158 | ====================== 3CX SmartSBC 16.2.24 @ ch-3cx-sbc ======================
CRIT | 20201202-004852.241 | 3CX | SBC | 1995921936 | Log.cpp:159 | Log started: type = file, level = ERR, size = 104857600, resip = ERR, remote = -1
CRIT | 20201202-004857.331 | 3CX | SBC | 1995921936 | Bridge.cpp:28 | System Clock: resolution 0 microseconds, not monotonic
ERR | 20201202-004857.337 | 3CX | RESIP | 1995921936 | ssl/Security.cxx:257 | Error calling stat() for /var/lib/3cxsbc/.sipCerts: No such file or directory
CRIT | 20201202-004902.103 | 3CX | SBC | 1995921936 | RPiTunnel.cpp:373 | Running in console mode

Is there more than one process that sends a version number to the server? Maybe one component didn't get updated?

I could try restarting the SBC...? Haven't done that yet.

 

[JohnS_3CX]

Given we did not find any evidence in the logs of this Pi, and that you see the version changing on your screen then I suggest you check the following 2 things:

1) When it happens again, open a Chrome browser window via incognito to eliminate any cache issues which may be showing you the wrong version. Just to be on the safe side

2) Check that there is definitely no rogue 2nd SBC running. The Local IP of the SBC that appears in your management console may appear to also change, indicating that there is indeed a 2nd SBC. I think this is one thing that needs to be made certain. The SBC may be on a PC, or a VM - not necessarily another Pi

 

[SteveITS]

I get where you're coming from. If I refresh the two pages repeatedly I see the version change on the SIP trunk page but if I have another page open to "edit" the SBC trunk (https://xxxx.il.3cx.us:5001/#/app/sbc_editor/KUsqQo68Ztcb) I am not seeing the IP change there. Is there another place to look?

While this is the location where someone did manage to start installing a 3CX server instead of the Windows client, somehow, I know no PCs have any SBC software installed as we are their IT and we can see installed software. They don't have any VMs and aren't sophisticated enough to set one up on their own, plus I can see no computers have Hyper-V installed.

Version change verified in a private window...I just press F5 repeatedly and watch it flip back and forth every few seconds.

So, quite a mystery. I may just restart it after hours, unless you have a reason not to, for further troubleshooting.

 

[ivank]

Hi SteveITS, try restarting 3CXTunnel service first when possible and see if it helps.
If the problem persists, your best option would be to switch PBX into Verbose mode temporarily and collect tunnel logs.

 

[SteveITS]

I should clarify this is a Linux server, 16 u6. I restarted 3CX PhoneSystem 01 SIP/RTP Tunneling Proxy on the server, and that seems to have fixed it. At least, after several minutes and 50+ page reloads the version is staying at 16.2. I'll watch for a while but hopefully that did it.

 

[abercoms]

I had exactly the same, would be OK for a day or so then com back and show previous version, first attempt was to refresh the install, didnt work, I then uninstalled the SBC from the Pi then reinstalled, problem been gone for 2 weeks now

 

[ivank]

I restarted 3CX PhoneSystem 01 SIP/RTP Tunneling Proxy on the server, and that seems to have fixed it.

Looks like a stuck dead connection with SBC in Tunnel which does nothing but updating the connection stats with stale data. Will be addressed in future releases.

 

[SteveITS]

stuck dead connection with SBC in Tunnel

It is always fun to discover the "edge" cases! It still shows 16.2 this morning after 20ish page reloads so it looks fixed. Thanks Ivan and John for the assist.