Hi Don
I have followed this thread and it seems along the way you are giving conflicting information. In some cases you say the PBX is on 192.168.98.120, then 192.168.99.120. I know you are not using real IPs, but relative to each other this makes it impossible to decipher your traceroutes and examples.
Firstly, there is no harm in displaying your internal IP addresses as they are and it would a lot more helpful if you do. Nobody can hack your system knowing your internal IPs unless they know your external IP and a lot more information.
Is your PBX in the voice subnet with your phones, or the data subnet?
When you setup the BOVPN on the Watchguard did you create two tunnels? I.E.
Site A:
Adata -> Bdata
Avoice -> Bdata
Site B:
Bdata -> Adata
Bdata -> Avoice
With Watchguard BO VPN, like most VPN implementations on Linux based routers, you must create tunnels between any subnets that will communicate. You cannot do this with any form of routes. You need BOTH tunnels.
If your PBX is indeed in the voice VLAN, I would guess that you have done the tunnels correctly, or your phones could not even register. If the PBX is in the data VLAN, then your phones could register, but (in a normally setup system without modified options such as PBX delivers audio and SBCs, etc) the phones will try to communicate directly with each other, so there needs to be a way for phones to communicate directly with phones. This in itself should cause no way voice, but there may be other issues at play also.
You need to Wireshark this to see what is going wrong. At the moment you are assuming that there is a routing issue. I have setup hundreds of Watchguards with and without BOVPN. If you have the tunnels right and you have as you say anybovpn to anybovpn, any port and it is high enough in the rule list so it always gets hit, you should be fine.
Here are some points to note with Watchguard (similar to many other routers/firewalls).
Watchguard uses a single policy in both directions. Depending on which direction traffic starts, the policy that evaluates to that traffic is the policy that will carry the data in both directions. In your case, the any bovpn to any bovpn is the only policy, but it is an important oft misunderstood point.
As of about a week ago, the IPS signatures in Watchguard will interfere with SIP ACK (block it), so make IPS is not ticked on in your any bovpn policy. It won't affect your current issue, but it will cause other problems, so since we are talking about Watchguard, save yourself some pain.
You can packet capture in the Watchguards from FSM, or the CLI, but I would suggest you first run Wireshark on your computer. If you installed full graphical Debian you can run Wireshark as easily as on a Windows PC. If not you can either use TShark (text version), tcpdump (similar to TShark), or you can install XLaunch (XWindows server) on a Windows PC and run Wireshark on your Linux box through XLaunch to your PC screen. This is how I run Wireshark on client Linux boxes and from my Linux VMs in my own office.
When you run Wireshark, you can then use the amazing VOIP analysis tools to look at the call flow and see what IP addresses and ports are negotiated for the RTP. You can also see if the RTP stream is even leaving site B and whether it has the correct address and port in its destination.
Speculating on what might be happening without a trace is a pointless and frustrating (for you excercise). Until you can see the traffic you can't fix it.
Regards
Mark