15.5.400 cannot answer calls via 4G on Pixel 2

[Mahomed]

APP_VERSION:
1505856

PACKAGE_NAME:
com.tcx.sipphone14

PHONE_MODEL:
Pixel 2

ANDROID_VERSION:
9

BRAND:
google

PRODUCT:
walleye

BUILD_CONFIG:
APPLICATION_ID=com.tcx.sipphone14
BUILD_TYPE=release
DEBUG=false
FLAVOR=
VERSION_CODE=1505856
VERSION_NAME=15.5.400

Server Version: Standard 15.5.15502.6
Server OS: Debian

The above info was generated from "Send Logs". Please let me know if you need any further info.

Problem: Cannot answer calls on Android App on Pixel 2 running Android 9.0.0.

Basically if you press the answer button, it will stop ringing in the app, but the call will still continue ringing on the server. If I change the RTP mode from "Normal" to "Allow Secure/Only Secure" then the app can answer and hang up the call, but the caller will still hear the ringing tone.

Outbound calls from the app works fine. Only answering calls is the issue

This is not a SIP Trunk/Codec issue as indicated in this other thread https://www.3cx.com/community/threads/android-google-pixel-2-not-accepting-calls.60484/ because the mobile app can answer calls when it is on the office WiFi (same LAN as the 3CX server). The problem also occurs if the mobile is on 4G and an internal extension tries to call the mobile extension. (e.g. yealink ext 300 calls mobile extension on 305)

The issue seems to be specific to the Pixel 2 because the app is working fine on Samsung Note 8 (using Android 8) and a Pixel 3 XL (using Android 9). Also I've loaded the account that works fine on these phones onto the Pixel 2 and the problem continues. If you take the account from the Pixel 2 and provision it on the Note 8 or Pixel 3 XL, then it works fine.

I have tried several other things including:

• Uninstall the client, reboot, install the client from the store
• Change RTP mode from normal to "Allow Secure" or "Secure Only" and this allows the app to answer and hang up the call, but the call is not connected
• Set "PBX delivers Audio"
  
• Changed the order of the codecs - no difference
• Rebooted firewall
• Run firewall checker and passed
• Rebooted 3CX server
• Changed the transport mode to TCP, UDP and TLS
• Disabled/Enabled Push notifications
• Tried with and without the 3CX tunnel (app registers, the issue is only when trying to answer calls)
• Disabled NAT helper on the mobile
• I have made over 50 calls with different permutations of the above

I can see from
https://www.3cx.com/blog/change-log/3cxphone-android-build-history/ that the Pixel 2 is specifically mentioned but it doesn't say whether this is with Android 8 or 9.

It would be great if anyone else could confirm/check this and hopefully come up with a solution or let us know if it's a "cannot/won't" fix type situation.

Please let me know if you need any other info. Thanks.

 

[datadunn]

Hi.

Apologies for being concise but I am writing this on mobile.

We have being using 3CX successfully for 2 years. Our maintenance renewed on 31/10 this year and around the same time we start with the exact issues you are describing above. perhaps a coincidence and this actually align with the time of an update.

I have also troubleshooted and tested all settings similar to you. we work out of office alot and have come to rely on 4G so this is currently crippling us as a business.

we are running 4 x Samsung S9+. which worked fine April to October... this is the main difference between our scenarios.

I have also noticed it appears to work on any wifi. Not just the wifi for the lan with the server.

only 4g has the issue answering. no exceptions to this. havent been able to answer on 4g all month.

Kind Regards
James

 

[LeonidasG_3CX]

Can you try disabling the Improved Bluetooth & Audio Routing feature that's located in the Settings > Advanced part of the app?

This is enabled and whitelisted by default for the Pixel 2 as it's a supported device for this functionality. Try to do this and see if it helps or not.

 

[Mahomed]

Can you try disabling the Improved Bluetooth & Audio Routing feature that's located in the Settings > Advanced part of the app?

This is enabled and whitelisted by default for the Pixel 2 as it's a supported device for this functionality. Try to do this and see if it helps or not.

Unfortunately this has not made any difference. I have a verbose log from the app if that helps.

 

[Mahomed]

I have now tested this on an HTC and it works fine.

PHONE_MODEL:
HTC 10

ANDROID_VERSION:
8.0.0

It does not work on Huawei P20 Pro running Android 8.1.0.

On a hunch, I swapped the SIM cards around and it appears that the issue follows the SIM card. The SIMs that are directly on the EE network do not work. Users on other networks work fine regardless of the phone.

I suspect that calling EE is going to be a complete waste of time. Hopefully 3CX can help here?

@datadunn is there any possibility you could try a different SIM in your phones? I don't know what country you are in, but if in the UK like me, it would be worth a try.

 

[datadunn]

I have now tested this on an HTC and it works fine.
On a hunch, I swapped the SIM cards around and it appears that the issue follows the SIM card. The SIMs that are directly on the EE network do not work. Users on other networks work fine regardless of the phone.

I suspect that calling EE is going to be a complete waste of time. Hopefully 3CX can help here?

@datadunn is there any possibility you could try a different SIM in your phones? I don't know what country you are in, but if in the UK like me, it would be worth a try.

Very interesting - Thanks for that.. Yes I am in the UK. All 4 of our Samsung 9+ are on EE sim cards and all other phones are also EE. I have no other network available to test.

 

[datadunn]

I will start my own thread rather than Hijack yours as support may ask for info...
but @Mahomed see this post - the title is misleading as it seems to be similar. https://www.3cx.com/community/threads/3cx-and-ios-issues.60493/

They all claim that Android was working and only IOS was having issues on EE... However, I can verify android is also experiencing issues. It appears it actually may have something to do with IPv6 implementation at EE and the sim moving away from IPv4.

The timeline also perfectly aligns as I said i started seeing issues around the 30/10/2018

Kind Regards,
James

 

[datadunn]

Having just spoken to EE I can confirm they have been moving sim blocks completely away from IPv4 onto IPv6 and it seems our block was done at the end of October. It would seem 30%+ of all EE customers are now on IPv6 as the migrations continue.

It would appear you have also been migrated Mahomed.

@LeonidasG @Mahomed Looks like I need look into how to make the 3CX Server [Windows 10] talk on IPv6 as well... perhaps on a per extension basis as the network adapter for the mobile clients..

Thanks for pointing me in the right direction on the EE front and getting us there. Looks like now we are aware of the issue.

Kind Regards,
James

 

[LeonidasG_3CX]

We've done a lot of work related to IPV6 in the last releases of the iOS and Android clients. Any feedback on how well or bad it works on IPV6 is always welcome.

 

[Mahomed]

Thanks for posting back with the extra info. I guess their v6-to-v4 translation is breaking something along the way. We did a port scan from the mobile and that worked. So it just seems to be failing to translate some of the packets correctly. Probably an MTU size issue somewhere on their network I'd expect.

 

[Steven Fletcher]

EE does NAT64 - I am in the same situation as you describe - my phone rings and I can answer calls whilst on 4G (and hear nothing), but the caller still hears ringing/hold music.

For my system, there are no internal users, only external Android/iPhone push clients.

My 3CX phonesystem runs in AWS and only registers a DNS record at 3cx.co.uk using an IPv4 'A' record, despite having an IPv6 address on the Debian server (dual stack)

The provisioning file/configuration only mentions the 3cx.co.uk DNS address, not a specific IPv4/v6 address.

What seems to be happening is this:

1. Call lands on the 3CX system
2. Caller hears ringing or hold music
3. Push notification is sent from 3CX to the Android device
4. 3CX (Android) client on EE 4G does a DNS A/AAAA lookup for <phonesystem>.3cx.co.uk
5. EE's DNS64 looks for an AAAA and A record for <phonesystem>.3cx.co.uk
6. 3CX DNS servers answer with an IPv4 A record
7. EE translate the returned IPv4 address to a NAT64 address beginning 64:ff9b::something
8. 3CX (Android) client genuinely believes that the IP address of the 3CX server is 64:ff9b::something, but it's not.
9. IPv6 traffic to 64::ff9b::something passes through EE NAT64 and comes out as IPv4 traffic from EE's NAT addresses (e.g. 213.205.241.202)
10. SIP traffic from the NAT address lands on the 3CX server's IPv4 address, but the headers contain Via's with 64:ff9b::something IPv6 addreses
11. Some conversation occurs
12. SDP packets try to negotiate an IPv4 media path from the server to the client at "192.0.0.4"

(192.0.0.4 is the IPv4 address that EE issue to all IPv6 464XLAT users, in other words, all 4G users on a device that supports IPv6)

I suspect this is causing "Malformed SIP packet" errors in the 3CXTunnel logfile, but I'm not sure whether that's directly causing any problem with the media path.

The Android client thinks it's at IPv4 address 192.0.0.4, when it's not really: that's an EE IPv4 NAT address.

I've tried forcing the Android client to use the IPv6 address by tapping it into the configuration. The SIP conversation seems to work OK, but the SDP media still attempts to use 192.0.0.4.

A couple of suggestions then...

1. 3CX dynamic DNS should support IPv6 addresses
2. As a workaround (for now) there should really be some option in 3CX to force IPv4 for all communication paths (DNS lookup, SIP conversation, media path)
3. The 3CX Android client should be wary of conflating IPv4 SIP with IPv6 SIP. If the SIP conversation is IPv6 then the SDP from the should *really* use the client's real IPv6 address...

 

[Steven Fletcher]

Having just spoken to EE I can confirm they have been moving sim blocks completely away from IPv4 onto IPv6 and it seems our block was done at the end of October. It would seem 30%+ of all EE customers are now on IPv6 as the migrations continue.

https://indico.uknof.org.uk/event/38/contribution/8/material/slides/1.pdf will explain the logic behind EE's migration.

 

[datadunn]

https://indico.uknof.org.uk/event/38/contribution/8/material/slides/1.pdf will explain the logic behind EE's migration.

Yeah I understand the need behind it... I just wasn't aware of the transition and probably wouldn't have foreseen this issue anyway.... Now I just need to overcome this hurdle as I have an IPv4 only ISP and I host on Windows 10 locally in my office and not on AWS.

 

[Mahomed]

Similar situation here. Our ISP can give us IPv6 but we would have to upgrade our firewall and re-configure our internal network.

A workaround would be helpful to buy us (and many others affected) some time.

 

[Steven Fletcher]

I can see many 3CX users are going to be in this situation, considering EE has such a huge corner of the UK 4G market.

For the record - I know for a fact that my EE device has been using IPv6 464XLAT for at least 18 months. It's not an new EE problem for me.

It's only recently, since 3CX app/server has been moving to use IPv6 more, that I've had trouble with calls on EE 4G. Given the relative infrequency of calls on my device, I just assumed it was something I was doing wrong, or that it an issue with my client device, server OS, or some other connectivity blip, rather than anything else!

Because my server can support IPv6, my quick workaround - if I knew for certain it was going to work - would be to stop using 3cx.co.uk DNS, and move to a static hostname that supports IPv4 and IPv6 addressing.

The trouble is, moving DNS to another provider is not ideal for the public cloud service that I'm using, and it definitely isn't a quick or easy option for others like yourselves who don't have native IPv6 connectivity to their server.

The 3CX client really needs to understand that connectivity between it and the server might be tampered with, as a necessary evil of moving the world to IPv6.

 

[Steven Fletcher]

Here's a really ugly workaround if you can't wait for a 3CX modification, I've just tried it and it seems to work for me.. but I hate it: force your mobile device APN to use IPv4 only.

Your mileage may vary, you might break other apps, it might not work in the future, and (urgh) you're intentionally not using IPv6, but it might be worth a shot for now.

You'll probably need to add a new APN and choose that, rather than try to modify your provisioned APN settings (https://community.ee.co.uk/t5/4G-and-mobile-data/EE-APN-settings-Where-to-find-them/td-p/132083 has the settings).

On Android/Samsung - set "APN Protocol" to IPv4 only.

Verify that it's worked by checking your "real" IP address - look at your phone's diagnostics screen - you should see only a 10.x.x.x address, and not a 2a01:4c8:: & 192.0.0.4 address.

 

[datadunn]

You'll probably need to add a new APN and choose that, rather than try to modify your provisioned APN settings (https://community.ee.co.uk/t5/4G-and-mobile-data/EE-APN-settings-Where-to-find-them/td-p/132083 has the settings).

@Steven Fletcher Thank you very much for that work around. I had been googling a way to try and do this but couldn't find a working solution as of yet. I also asked if there was a solution like this to both of the EE techs that I was escalated to and was assured that there was no way to make my SIM get an IPv4 address now that our plans had migrated.

I can definitely see this becoming a problem like you said... especially with smaller clients and any that self host.

I have gathered loads of data and done so much testing but where our phones seem fail in the process when capturing the streams, etc.. was the following;

Via: SIP/2.0/UDP [::ffff:213.205.192.244]:19816;branch=z9hG4bK-524287-1---tunneltid;rport;

ACK / Invite SDP / Trying / Ringing all get through but the "Answer" OK SDP never makes it back to the server.. the call with perpetually stay in ringing until timeout.

Thank you again for the APN workaround. I will run that for now and continue to have a play.

 

[Steven Fletcher]

Your Via: header differs slightly from mine. The one you've quoted has a IPv4 address socket - is this definitely the one that's causing you trouble? The 213.205.192.244 address is one of EE's CGNAT addresses. My Via: headers have DNS64 addresses in them.

The fundamental issue (as far as I can see) is that the SIP packet containing the SDP is malformed, likely because the multiple mentions of the IPv6 NAT64 address aren't properly encapsulated by [ and ].

Because it's malformed, it's likely that 3CX can't make use of it, so can't ever use the information to setup the media path.

22:05:03.938|7f3f73658700|Error|/home/repomaster/workspace/15.5SP6/SPBuild/Sources/Projects/Tunnel/SipCore/SipPacket.cpp(41): Malformed SIP packet: size=1406:
SIP/2.0 200 OK
Via: SIP/2.0/UDP 64:ff9b::1234:5678:5090;rport;branch=3cxtnl-iPhoneTID-1
Via: SIP/2.0/UDP 127.0.0.1:5060;rport=5060;received=64:ff9b::1234:5678;branch=z9hG4bK-524287-1---a1e5ef79f74e1272
Call-ID: 6498106c-ded3-4b13-8757-cfdc6d7ad7e7
From: "Meublic" <sip:[email protected];tag3cx=352c9b2d-2a1d-47bc-b501-e806b273c13b>;tag=9ec2140f
To: "My Extension" <sip:[email protected]>;tag=d9861011-b90e-42f0-b7dc-8c407f4fe305
CSeq: 2 INVITE
Session-Expires: 1800;refresher=uas
Contact: "My Extension" <sip:[email protected]:5060;inst="c2b8a198";rinstance=1-17751d3a-e7f4-424c-b90d-9f1f57209d06>
Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS
Supported: replaces, 100rel, timer, norefersub
Content-Type: application/sdp
Content-Length: 561

v=0
o=- 3752431502 3752431503 IN IP4 192.0.0.4
s=pjmedia
b=AS:117
t=0 0
a=X-nat:0
m=audio 10000 RTP/AVP 98 97 99 104 3 0 8 9 120 18 96
c=IN IP4 192.0.0.4
b=TIAS:96000
b=AS:117
a=rtcp:10001 IN IP4 192.0.0.4
a=sendrecv
a=rtpmap:98 speex/16000
a=rtpmap:97 speex/8000
a=rtpmap:99 speex/32000
a=rtpmap:104 iLBC/8000
a=fmtp:104 mode=30
a=rtpmap:3 GSM/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:9 G722/8000
a=rtpmap:120 opus/48000/2
a=fmtp:120 useinbandfec=1
a=rtpmap:18 G729/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-16

 

[Mahomed]

Thanks for all the extra info Steven. I'll see if we can do the IPv4 workaround when my colleagues get in.

@LeonidasG given all the above information, do you think 3CX now has enough information to have the devs take another look at this please?

 

[Mahomed]

I can confirm that changing the APN has worked for us. As we only need to use the client occasionally outside the office, this solution will work until 3CX can fix this and/or we get IPv6 native in the office.