AMAZON CHIME OUTBOUND 408 ERROR

[jrgdnn12]

Hello

I am getting and error message about 6 times sin 3 days. Do you know what it means?

Call or Registration to +1702*******@(Ln.10000@Amazon Chime Voice Connector) has failed. 0.0.0.0 replied: 408 Request Timeout; internal
SIP Server ID: 12294 01/25/2021 10:08:44 PM

I am also awaiting a reply from amazon aws support.

Thanks

 

[greatchoice]

Please note that i deleted the capture as it contained sensitive information. Wireshark captures are not meant to be shared in public forums so please try to avoid doing that in the future.

From the capture it looks like the call is being sent to AWS but there is no reply. Eventually the PBX switches to the secondary route.

Can you try something and let me know if that makes any difference?

Go to your AWS Chime trunk settings and navigate to the Options tab. Scroll down to the Transport Protocol option and change the value to UDP.

View attachment 20419

Press OK to save your changes and try again. You must also access your AWS account and make the change there under the Origination tab so that you can also receive calls via UDP.
Let us know how that goes.

Very interesting here. I did notice that mine was using TCP as default. I was not having these issues but I did change it to UDP as SIP is UDP!

 

[NickD_3CX]

but I did change it to UDP as SIP is UDP!

Ok, not exactly, SIP works over TCP and TLS, not only UDP, but the point is, if this helped with your problem, that is good news!

FYI, we tested with TCP and we didn't have a similar problem and prefer it as it is a more reliable transport.

 

[jrgdnn12]

it’s been a few days since the switch to UDP. And I have not experience the problem. AWS just says they are continuing to work on it.

etiher AWS fixed something to UDP Is keeping the connection alive.

Hope not to see this problem again.

 

[jrgdnn12]

I wanted to check in to see if those having this problem are using Amazon EC2 hosts as we are.

Mine is hosted only lightsail. Same ec2.

 

[jrgdnn12]

A word from AWS Support

We have just received an update from our service team and they have provided the following comments: "In the example provided, a call failed because it was sent to 3.80.16.103, which - at the time, was not one of the active edge proxy hosts. The IP addresses that a Voice Connector outbound host name resolves to may change; caching responses for longer than the DNS TTL or sending calls directly to an IP address vs. the outbound host name may result in call failures. Please see the first note in https://docs.aws.amazon.com/chime/latest/ag/edit-voicecon.html for additional details." I hope this helps, but please let us know if the issue persists. We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center. Best regards, Tashreeq J. Amazon Web Services

Signaling through UDP might help in this case. But if TCP is superior. A patch is needed from 3CX.

 

[YiannisH_3CX]

That is interesting as i would have never expected Amazon to switch servers on and off. In theory if this is what is happening then UDP should have the same issue as well as 3CX would try to send the Invite blindly to an IP that does not listen for Invites.

The difference between TCP and UDP is that in TCP the PBX will try to establish a TCP connection first before sending the Invite. If that fails then the PBX will move to the next IP. UDP will send the Invite anyway and wait for a reply. What i would expect to see is Amazon terminating the TCP connection when the server goes offline.

Both approaches though would wait for 32 seconds before moving on to the next server so in theory from a users perspective the issue would look the same.

If you are interested in doing some additional testing please let me know so i can p.m. you to arrange something.

 

[jrgdnn12]

If you are interested in doing some additional testing please let me know so i can p.m. you to arrange something.

Yes, but I only have a production deployment. please let me know.

thank you

 

[YiannisH_3CX]

Yes, but I only have a production deployment. please let me know.

thank you

Great, I will send you a p.m. shortly.

 

[JPoro]

I am a new (3 months in testing) user here and I've been frustrated with this same issue...I can test if you need.

Intermittent outgoing calls
No logging on Amazon's side. (just changed that to on)
TCP was selected...just changed to UDP

We'll see if it works now or not

 

[robinsonconsulting]

I've changed all servers to use UDP outbound and have not seen that error again yet.

 

[jrgdnn12]

Happened again today. Refreshing registration fixed the problem immediately.

PS. I had switched on UDP

 

[robinsonconsulting]

Just saw the same thing on one of my trunks as well. What a pain!

 

[YiannisH_3CX]

Just saw the same thing on one of my trunks as well. What a pain!

Did this happen while on UDP?

 

[robinsonconsulting]

Yes it did and it seems like it only happens Monday mornings.

 

[YiannisH_3CX]

Yes it did and it seems like it only happens Monday mornings.

That is probably the trunk is idle during the weekend and the IPs change. We are investigating this issue internally and we are trying to replicate to see what is causing and how we can improve it but its proving difficult to replicate since it happens randomly .

 

[JPoro]

Still NOT working!!! Didn't work today, had to "refresh registration" even while using UDP.

 

[cobaltit]

Still NOT working!!! Didn't work today, had to "refresh registration" even while using UDP.

Could always switch to a different provider then

 

[YiannisH_3CX]

I have good news and bad news. The good news is that we partially managed to replicate the issue. The bad news is that our DNS checks tell us that the AWS that was not responding was still in the list of A records provided by the DNS check .

This could mean a couple of things, the first is that the server had an issue at that time and was not responding without being taken out of the rotation so we actually did not replicate the actual issue. Second is that the server was taken out of the rotation but our DNS server had not yet propagate the correct results.

You can always force the PBX to work by leaving an outbound call to timeout for 32 seconds. Once the 32 seconds elapse the next call will move to the next DNS target.

To be able to better troubleshoot this we need some assistance from people that are facing the issue. We have a command that will tell the machine to perform a DNS check every few minutes and write them to a file.

while :; do date; dig +short a [Your_unique_id].voiceconnector.chime.aws; sleep 180s; done > /tmp/dns_lookup.log

This will perform a DNS request for your unique registrar every 3 minutes and save them to a file for later. You will need to replace the [Your_unique_id] with your unique value that you used as a registrar for your trunk.

So when the issue occurs and without doing anything else start a capture from the management console --> Dashboard / Activity Log / Select all interfaces and start the capture. Try to make a call. Once it fails stop the capture and save it. Send me a p.m. with the capture and the dns_lookup.log file so we can see if at the time that the issue occurred the target IP was not in the DNS list.

 

[YiannisH_3CX]

One important thing to add is that the terminal needs to be open for the command to work and keep running so i would recommend once the issue appears run the command in the terminal, wait for 5 minutes, run a few calls and connect the data so that you do not leave the terminal running

 

[jrgdnn12]

That is probably the trunk is idle during the weekend and the IPs change. We are investigating this issue internally and we are trying to replicate to see what is causing and how we can improve it but its proving difficult to replicate since it happens randomly .

Our trunk is never idle. The problem occured once in the past month.