3CX SMTP Blacklisting

[Lee Cramman]

We have a voicemail box where customers leave medication orders for us to arrange for them. Voicemails are forwarded by email to a shared mailbox used by the relevant team, 3CX then deletes the forwarded email. The mailbox is very well used - it can receive hundreds of messages per day. It all works very well. Usually...

My problem is that if there are any sort of problem with the receiving mailbox (say a shortish network outage) and a small number of message bounce, 3CX appears to blacklist that email address (for ? amount of time) and, during the blacklist, the voicemail emails just disappear into a big black hole. The 3CX SMTP server doesn't appear to ever retry to send. All the while these emails are disappearing, 3CX still happily deletes the voicemails from the voicemail mailbox as successfully sent... so the messages are lost forever.

This is obviously far from ideal. It's happened again today and we've lost about 2 hours worth of orders. We either have to reconstruct who called us from reports / logs and call every single one of them back to get the orders again or, in a couple of days, we will have some very angry patients who don't have their medication.

Due to the volume involved, keeping the voicemails in 3CX isn't really an option - we do need to email them. A delay is acceptable but for messages to simply disappear into the ether is not.

Is there any way to stop an email address being blacklisted so that this doesn't happen in the future?

Also, is there any way 3CX could look to change this behaviour? The voicemails that can't be emailed staying in the voicemail mailbox would be acceptable or messages being resent a couple of hours later would also work.

 

[SweetAction]

It's just happened again... No alert letting us know there is a blacklisting problem, no way of unblacklisting, no way of knowing duration of auto-ban, emails / voice attachments during the auto-ban will disappear into a black hole rather than be re-sent later on...

Luckily, after last time we've set things up up to retain a certain number of voicemails for that extension but spotting the problem in the first place relies on someone noticing that traffic has disappeared and logging in to the dummy extension and downloading thousands of voicemails over ?period is onerous.

I really don't understand why you're still fighting with this.

What email system do you use for normal communication? When someone emails you, what mail server gets that email? Why can't you use that system INSTEAD of 3CX SMTP?

 

[SteveITS]

because (presumably) Google had a brief outage on their service, the 3CX smtp server blacklisted the Gmail address

If you are using Gmail for SMTP then 3CX's SMTP service is not involved.

I think these are both for Workspace? (needs app password):
https://support.google.com/a/answer/9003945?hl=en
https://support.google.com/a/answer/176600?hl=en

Not sure about free Gmail...in a quick search, looks like they might limit that to 500 emails/day?
https://kinsta.com/blog/gmail-smtp-server/

 

[Lee Cramman]

If you are using Gmail for SMTP then 3CX's SMTP service is not involved.

I think these are both for Workspace? (needs app password):
https://support.google.com/a/answer/9003945?hl=en
https://support.google.com/a/answer/176600?hl=en

Not sure about free Gmail...in a quick search, looks like they might limit that to 500 emails/day?
https://kinsta.com/blog/gmail-smtp-server/

We are not using Google as 3CX smtp server, I've never said that. We are sending using 3CX smtp server to a gmail address in the hope that it doesn't get blacklisted as easily.

smtp is the means by which 3CX sends. Our corporate email system is what staff use to send and receive. 3CX sends, our corporate system is offline for a few minutes or there is an internet outage somewhere and the email address immediately ends up blacklisted with 3CX as it "Doesn't exist". This is not normal behaviour for an smtp server, you would expect a delay and then one or more retries with a final failure after a certain period (usualy hours or days) unless the receiving server responded with a "User not found" in which case there would be no retries and a (correct) blacklisting by 3CX.

We have zero scope to spin up a VPS and run our own mail server, we have to use the 3CX option. Our existing corporate email is part of the UKs NHS. As a government / health system it's locked down extremely tight, we have no direct control over it and no scope to utilise it for smtp in 3CX. Therefore, implementing alternative smtp ourselves, solely to work around the problem for a single mailbox, is simply not an option.

As a different workaround we tried using 3CX smtp to send those emails to a (not free) gmail address which in turn auto forwards them to corporate email. The idea being gmail = high availability / uptime so less likely to get incorrectly blacklisted. This proved more reliable (it's gmail so outages are few and far between) but occasional outages can still happen, at which point... the gmail address is blacklisted by 3CX smtp and around we go again. The problem is now less common but still exists.

tl;dr - It appears 3CX smtp does not behave as expected, we can not work around that by using our own smtp, it would be completely impractical for us to implement an entire new email solution for a single mailbox.

I really don't understand why you're still fighting with this.

What email system do you use for normal communication? When someone emails you, what mail server gets that email? Why can't you use that system INSTEAD of 3CX SMTP?

See above - we are required to use NHS Mail for corporate email. It's a huge corporate systen (>1m users) and is locked down tighter than tight, we do not run, own or control it and are therefore unable to request direct use of smtp for our PBX.

I'm still fighting with it as the only answer seems to be to implement smtp elsewhere which is not practical for us.

And, to be fair, I've pretty much given up fighting - I returned to the thread to have a frustrated moan about how 3CX implement smtp and the risk the approach represents to potentially important data.

My final workaround has been to retain the voicemails in 3CX in case this happens so we always have a backup. This is clunky but works, however it is often hours before anyone notices that the stream of voicemails has dried up and we have been blacklisted again leading to a backlog. Better than losing all voicemails during the blacklist period (as previously happened) but not ideal.

 

[SweetAction]

See above - we are required to use NHS Mail for corporate email. It's a huge corporate systen (>1m users) and is locked down tighter than tight, we do not run, own or control it and are therefore unable to request direct use of smtp for our PBX.

I'm still fighting with it as the only answer seems to be to implement smtp elsewhere which is not practical for us.

And, to be fair, I've pretty much given up fighting - I returned to the thread to have a frustrated moan about how 3CX implement smtp and the risk the approach represents to potentially important data.

My final workaround has been to retain the voicemails in 3CX in case this happens so we always have a backup. This is clunky but works, however it is often hours before anyone notices that the stream of voicemails has dried up and we have been blacklisted again leading to a backlog. Better than losing all voicemails during the blacklist period (as previously happened) but not ideal.

Ok, that explains why you use 3CX SMTP over your own mail server.

Would you be willing to pay $15 USD per month to avoid this? Instead of spinning up and managing your own SMTP server, you can use a transactional service like SMTP2Go as your SMTP service. Pay monthly, someone else handles the mail server.

 

[SteveITS]

@Lee Cramman Ah, I misinterpreted "tried using Gmail to relay the messages" as "tried using Gmail for SMTP relay."

We relay our own through an M365 connector so I can't easily look at message headers to see which company 3CX uses. FWIW SMTP2Go says they block on hard bounces but not soft bounces.

 

[SweetAction]

We relay our own through an M365 connector so I can't easily look at message headers to see which company 3CX uses.

Sendgrid

 

[SteveITS]

@Lee Cramman I accidentally tested this for you using a custom SMTP server. I had tried enabling "Enable SSL/TLS" and thought I disabled it since using 365 as SMTP server needs it off. Apparently unchecking it didn't take, since I found I received a voicemail an hour ago with no notification. I unchecked the setting, and my voicemail notification from an hour ago was delivered.

Edit: at some point they do expire because we found one from last week that didn't get sent.

 

[Lee Cramman]

Ok, that explains why you use 3CX SMTP over your own mail server.

Would you be willing to pay $15 USD per month to avoid this? Instead of spinning up and managing your own SMTP server, you can use a transactional service like SMTP2Go as your SMTP service. Pay monthly, someone else handles the mail server.

That's a much better price than I've seen elsewhere! Brevo wanted $42 per month for an account with a dedicated IP, which is more than we pay for 3CX hosting.

I'll give them a look, thank you! I could probably save more than the cost by switching to them for our transactional emails (currently on a Brevo account with a non-dedicated IP which is sometimes temporarily blacklisted by Trend and costing about $21pm).

edit: A pro account (with a fixed IP, which is necessary if you don't want to end up on the Trend QIL now and then) is $75 per month...

@Lee Cramman I accidentally tested this for you using a custom SMTP server. I had tried enabling "Enable SSL/TLS" and thought I disabled it since using 365 as SMTP server needs it off. Apparently unchecking it didn't take, since I found I received a voicemail an hour ago with no notification. I unchecked the setting, and my voicemail notification from an hour ago was delivered.

Edit: at some point they do expire because we found one from last week that didn't get sent.

thanks for the info Steve.

 

[SweetAction]

That's a much better price than I've seen elsewhere! Brevo wanted $42 per month for an account with a dedicated IP, which is more than we pay for 3CX hosting.

I'll give them a look, thank you! I could probably save more than the cost by switching to them for our transactional emails (currently on a Brevo account with a non-dedicated IP which is sometimes temporarily blacklisted by Trend and costing about $21pm).

edit: A pro account (with a fixed IP, which is necessary if you don't want to end up on the Trend QIL now and then) is $75 per month...

You don't need a dedicated IP. Can you explain why you want a dedicated IP?

 

[Lee Cramman]

You don't need a dedicated IP. Can you explain why you want a dedicated IP?

My experience is that using a shared IP can leave you temporarily blacklisted via Trend QIL on the receiving end (rather than the sending end) due to someone else on the same IP spamming. We currently use a non-dedicated IP for transactional emails with Brevo and it's blacklisted from time to time.

It usually only lasts a few hours as Brevo support quickly jump on it and update Trend, but it's another potential outage.

A dedicated IP means you can manage your own email reputation.

 

[SweetAction]

My experience is that using a shared IP can leave you temporarily blacklisted via Trend QIL on the receiving end (rather than the sending end) due to someone else on the same IP spamming. We currently use a non-dedicated IP for transactional emails with Brevo and it's blacklisted from time to time.

It usually only lasts a few hours as Brevo support quickly jump on it and update Trend, but it's another potential outage.

A dedicated IP means you can manage your own email reputation.

I'd argue that any mail filter making decisions based on the IP itself is a bad mail filter these days. I mean, use it as a data point, but look at the email and make decisions. Just saying "it's on the RBL, block" is.... very old school.

Put another way, if you are using 365 or GSuite, you don't have a dedicated IP. They are sending out mail via a IP Pool - half the IPs are on some RBL or another at any given point. Most systems say "hey, it's 365, this happens" and grade the mail based on the contents, links, etc in the email.

 

[SteveITS]

@SweetAction Not Google inbound, though; I've been fighting this for a couple of weeks...M365 to Gmail often bounces with:

[email protected]
Remote server returned '550 5.4.300 Message expired -> 421 4.7.28 [2a01:111:f400:7e8a::719 15] Our system has detected an unusual;rate of unsolicited mail originating from your IP address. To;protect our users from spam, mail sent from your IP address has been;temporarily rate limited. Please visit; https://support.google.com/mail/?p=UnsolicitedRateLimitError to;review our Bulk Email Senders Guidelines.

 

[SweetAction]

I mean at the end of the day, even using 3CX SMTP doesn't mean the IP(s) won't end up on a block list. Even if 3CX is using dedicated IPs from Sendgrid, that IP can end up on a block list.

If this email is so critical, then maybe Brevo with a dedicated IP (while more expensive then the 3CX hosting itself) is the business need. You don't have many options here with all the limits and requirements you've laid out.

MailerSend (mailersend.com/) has a dedicated IP addon at $25/mo plan, might be cheaper.

 

[Lee Cramman]

I mean at the end of the day, even using 3CX SMTP doesn't mean the IP(s) won't end up on a block list. Even if 3CX is using dedicated IPs from Sendgrid, that IP can end up on a block list.

If this email is so critical, then maybe Brevo with a dedicated IP (while more expensive then the 3CX hosting itself) is the business need. You don't have many options here with all the limits and requirements you've laid out.

MailerSend (mailersend.com/) has a dedicated IP addon at $25/mo plan, might be cheaper.

Cheers, the longer this goes on the more I think you're right - yes, they probably are a bit less expensive.

 

[Lee Cramman]

I'd argue that any mail filter making decisions based on the IP itself is a bad mail filter these days. I mean, use it as a data point, but look at the email and make decisions. Just saying "it's on the RBL, block" is.... very old school.

Put another way, if you are using 365 or GSuite, you don't have a dedicated IP. They are sending out mail via a IP Pool - half the IPs are on some RBL or another at any given point. Most systems say "hey, it's 365, this happens" and grade the mail based on the contents, links, etc in the email.

The Trend QIL is very widely used... and normally works without causing too many issues as smtp retries take care of the temporary blacklisting at the receiving end (and, of course, systems do have differing rules for IPs belonging to businesses like Google / Microsoft with a high trust reputation). Which is where this discussion gets a bit circular - instead of a retry in a few hours (as most smtp services would) 3CX appears to blacklist at the sending end when this happens; which is the reason for this thread.

If i were a partner rather than a customer, I would definitely be posting a development suggestion about modifying this behaviour and enabling a system of smtp retries, although I can see why 3CX might not be keen to have huge numbers of messages stack up on a service they are paying for if / when there are major outages.

For now, the only options appear to be to pay for a fixed IP, high availability service or redesign our own processes to use an external mailbox from our SIP trunk provider.

 

[SweetAction]

While I'm not your partner, I can see why you want your request, so I made the idea thread for you: https://www.3cx.com/community/threa...s-3-times-before-blacklisting-address.122731/

I hope this helps, as I do not have anything further to add here. You're in a bit of a bind due to all the limits and requirements you have.

 

[Lee Cramman]

While I'm not your partner, I can see why you want your request, so I made the idea thread for you: https://www.3cx.com/community/threa...s-3-times-before-blacklisting-address.122731/

I hope this helps, as I do not have anything further to add here. You're in a bit of a bind due to all the limits and requirements you have.

Much appreciated! I've voted (and I see others have too). I owe you a pint!

Out of sheer frustration I've again approached our mail administrators to revisit the possibility of getting a corporate SMTP account (read: had a bit of a rant about patient safety to our customer liaison officer). This time the response was not a flat no but a "What's your use / business case... OK, we'll get back to you". Fingers crossed as that would sidestep all my problems given they're unlikely to blacklist an address on their own server.