3CX SMTP Blacklisting

[Lee Cramman]

We have a voicemail box where customers leave medication orders for us to arrange for them. Voicemails are forwarded by email to a shared mailbox used by the relevant team, 3CX then deletes the forwarded email. The mailbox is very well used - it can receive hundreds of messages per day. It all works very well. Usually...

My problem is that if there are any sort of problem with the receiving mailbox (say a shortish network outage) and a small number of message bounce, 3CX appears to blacklist that email address (for ? amount of time) and, during the blacklist, the voicemail emails just disappear into a big black hole. The 3CX SMTP server doesn't appear to ever retry to send. All the while these emails are disappearing, 3CX still happily deletes the voicemails from the voicemail mailbox as successfully sent... so the messages are lost forever.

This is obviously far from ideal. It's happened again today and we've lost about 2 hours worth of orders. We either have to reconstruct who called us from reports / logs and call every single one of them back to get the orders again or, in a couple of days, we will have some very angry patients who don't have their medication.

Due to the volume involved, keeping the voicemails in 3CX isn't really an option - we do need to email them. A delay is acceptable but for messages to simply disappear into the ether is not.

Is there any way to stop an email address being blacklisted so that this doesn't happen in the future?

Also, is there any way 3CX could look to change this behaviour? The voicemails that can't be emailed staying in the voicemail mailbox would be acceptable or messages being resent a couple of hours later would also work.

 

[SnakeRiverWebDevelopment]

If you are using Pro or Enterprise you can (not startup) you can use a custom SMTP server, this may be the way to go to prevent blacklisting.

 

[Lee Cramman]

Thanks, but spinning up a paid SMTP server / account somewhere (and the associated headaches of that) just to deal with the voicemails of a single extension seems like a bit too much of a side-trip for me.

Some clarity on what part of this is by design and exactly how the blacklisting process works would be a start. It simply seems really odd to me that there is a big black hole that this sort of thing can fall into.

 

[SteveITS]

I don't have inside knowledge here but usually the big SMTP services will automatically stop sending to failed addresses because they don't want to be seen as spammy. Now, one can reasonably argue "can't connect" is not the same as "address doesn't exist."

Are you on-premises? One can set up an in house SMTP server to relay out. Windows Server used to include this but it was removed in Server 2022 as I recall.

re: cost of SMTP, it shouldn't be that much for a regular hosting account that can provide email. I assume we're not talking about 200 per hour or something like that. Another option is to send to a mail server in your office that then relays out. There are free mail server programs out there still. Long (long) ago we used to use Mercury (pmail.com) which is paid. I've heard of but have not used hmailserver.com.

Overall it does seem like a design limitation that mail is not queued on their SMTP service, at least for a short time. Does anyone know if 3CX locally queues those messages, when using a custom SMTP server, for the case when the custom SMTP server is unreachable?

 

[Lee Cramman]

We're cloud hosted. These days the complexity of running your own relay just isn't worth it unless you have a significant IT function in the organisation. I could use a relay service but I don't see any guarantees that will stop this happening...

My understanding is that, in email, there are hard and soft bounces. A hard bounce (e.g. this account does not exist on this server) should = a fail and no more retries. For anything else, a few retries a few hours apart is reasonable (normal?).

You shouldn't get blacklisted for spam (e.g. Trend QIL) unless you're ignoring those hard bounces or your messages look really spammy.

 

[BrenttG]

We are also seeing that GMail and Office 365 have blocked 3CX's email server, none of our office 365 customers can get welcome emails or reports, etc, they don't even appear in the junk folder, they are just blocked within 365. Microsoft seems to be blocking them before they even hit the Tenant level.

Gmail ones do go to the junk folder and are marked as Dangerous


This has effectively made it a very big issue today for us as we were onboarding a customer, XD

 

[SweetAction]

Thanks, but spinning up a paid SMTP server / account somewhere (and the associated headaches of that) just to deal with the voicemails of a single extension seems like a bit too much of a side-trip for me.

Some clarity on what part of this is by design and exactly how the blacklisting process works would be a start. It simply seems really odd to me that there is a big black hole that this sort of thing can fall into.

You have email, right? Almost every service (365, GSuite, etc) allow you to direct relay to them and use them to send messages.

But the process is simple. 3CX takes the voicemail wav file, makes an email, sends it to their service, deletes it from the mailbox. It doesn't check that the service delivered it or not. I don't recall if they even check if the service accepted it - last I checked they didn't, but that was years ago.

The service basically takes any email that bounces and blacklists the address for 7 days. Adding/removing is automated. Bounce one message, it's blacklisted.

I've heard of but have not used hmailserver.com.

Overall it does seem like a design limitation that mail is not queued on their SMTP service, at least for a short time. Does anyone know if 3CX locally queues those messages, when using a custom SMTP server, for the case when the custom SMTP server is unreachable?

hmail works well.

3CX did not locally queue the messages last I checked (several years ago).

@Lee Cramman the truth is, it's a matter of how important this is. That will dictate how much you are willing to spend to resolve it. Using your own SMTP, with resend abilities, is not too expensive and will solve this. Run it in house but incur technical cost. Outsource that cost to a 3rd party but incur monthly cost. Whatever works best for you.

Alternatively don't delete the messages from the mailbox until you verify you have the email. You could even write some code to watch a shared mailbox and issue deletes to 3CX if you really wanted to.

 

[SteveITS]

don't delete the messages from the mailbox until you verify you have the email

A bit OT but IIRC it's not possible in StartUP to not attach & delete the voicemail.

GMail and Office 365 have blocked 3CX's email server

Didn't know about Gmail but M365 has been an open issue: https://www.3cx.com/community/threa...t-for-url-of-3cxphoneforwindows16-msi.121291/

 

[SweetAction]

A bit OT but IIRC it's not possible in StartUP to not attach & delete the voicemail.

Anyone who wants this level of control is not a good fit for startup.

Something something you get what you pay for

 

[Charles_3CX]

We are also seeing that GMail and Office 365 have blocked 3CX's email server, none of our office 365 customers can get welcome emails or reports, etc, they don't even appear in the junk folder, they are just blocked within 365. Microsoft seems to be blocking them before they even hit the Tenant level.

Gmail ones do go to the junk folder and are marked as Dangerous
View attachment 36162

This has effectively made it a very big issue today for us as we were onboarding a customer, XD

Thanks for reporting this and opening the ticket with us. We have replied to you on the ticket for more info when you can, please.

 

[SteveITS]

Anyone who wants this level of control is not a good fit for startup.

Something something you get what you pay for

Oh, concur. Just noting for the public. I have a list of limitations written down, that continues to lengthen. I'm sure it's a fit for some, but definitely not all.

 

[SteveITS]

A hard bounce (e.g. this account does not exist on this server) should = a fail and no more retries. For anything else, a few retries a few hours apart is reasonable (normal?).

You shouldn't get blacklisted for spam (e.g. Trend QIL) unless you're ignoring those hard bounces or your messages look really spammy.

Usually mail servers will retry every 30 minutes for 3-5 days, give or take. Often configurable.

The issue for the big services is they have a really high mail volume, for thousands of companies not just 3CX, so they really, really don't want to be flagged. It's very much a double edged sword. They're good at it, but have very low tolerance. I believe our ticketing system's service does something similar...it will not send to an address for I think 7 days if a message bounces.

From a programming standpoint, software like 3CX can either connect out right away, and succeed or fail, or it can drop the mail in a folder and retry for a while. A problem with the second option is the SMTP server may be misconfigured or unreachable, and the disk fills up. Every once in a while (in general, not specific to 3CX) a message gets stuck for some reason and sends out every 5 minutes, forever, every time the process on the server checks for queued mail. Perhaps 3CX could just delete any queued messages after 3 days or something, to work around that.

 

[BrenttG]

A bit OT but IIRC it's not possible in StartUP to not attach & delete the voicemail.


Didn't know about Gmail but M365 has been an open issue: https://www.3cx.com/community/threa...t-for-url-of-3cxphoneforwindows16-msi.121291/

It only affected us starting yesterday for 2 customers, today for several more we discovered when they contacted us when none of their daily reports came in from the 3CX Server, so i think the block took some time to roll through all of 365's segments. Yesterday we sent out some welcome emails and they seemed to go thru fine, but one customer couldn't get reports, then today when it snowballed we dug deeper, and here we are.

 

[Evolute IT]

Something something you get what you pay for

If you even pay for it

 

[Lee Cramman]

You havefffffff email, right? Almost every service (365, GSuite, etc) allow you to direct relay to them and use them to send messages.

But the process is simple. 3CX takes the voicemail wav file, makes an email, sends it to their service, deletes it from the mailbox. It doesn't check that the service delivered it or not. I don't recall if they even check if the service accepted it - last I checked they didn't, but that was years ago.

The service basically takes any email that bounces and blacklists the address for 7 days. Adding/removing is automated. Bounce one message, it's blacklisted.

hmail works well.

3CX did not locally queue the messages last I checked (several years ago).

@Lee Cramman the truth is, it's a matter of how important this is. That will dictate how much you are willing to spend to resolve it. Using your own SMTP, with resend abilities, is not too expensive and will solve this. Run it in house but incur technical cost. Outsource that cost to a 3rd party but incur monthly cost. Whatever works best for you.

Alternatively don't delete the messages from the mailbox until you verify you have the email. You could even write some code to watch a shared mailbox and issue deletes to 3CX if you really wanted to.

Apols for neglecting this thread - I've been on annual leave. We actually tried using Gmail to relay the messages. When there was a brief service outage with Gmail guess what? Blacklisted again...

I think your understanding of the current process is 100% correct. And one hard bounced message is a perfectly acceptable limit, abuse that and a global (e.g. Trend) blacklist would be completely justified - I wouldn't suggest 3CX change behaviour in this circumstance, getting globally blacklisted would be bad! A soft bounce though... or a mail server unreachable? By definition that's "Not deliverable at the current time" - to not retry that message later and instead blacklist the address for a week seems like odd behaviour to me (particularly the blacklisting when a server is unreachable). If it weren't blacklisted we might lose the odd message (still not great) but with blacklisting we can lose hundreds before the problem is noticed.

Your suggestion about not deleting messages from the mailbox is a valid one - you wouldn't even need to code, just set it to delete anything older than a week and set the mbox size to something that will never get full in that time. It still relies on us realising that stuff has stopped coming through though, which creates delay and uncertainty while we manually identify what has and what hasn't been processed / sent.

I think the only answer is, in fact, to not use email at all as, currently, there is too much risk of message loss and delay given the overly aggressive way 3CX handle blacklisting. I think I would say the same to everyone - voicemail via email is not reliable enough to handle critical messages without some sort of backup plan for when things inevitably go wrong.

 

[biggreyphone]

+1. Agree. Need notification that an email was shadow banned from 3CX. I didn’t receive attached VM for over a week until I realized.

 

[Lee Cramman]

It's just happened again... No alert letting us know there is a blacklisting problem, no way of unblacklisting, no way of knowing duration of auto-ban, emails / voice attachments during the auto-ban will disappear into a black hole rather than be re-sent later on...

Luckily, after last time we've set things up up to retain a certain number of voicemails for that extension but spotting the problem in the first place relies on someone noticing that traffic has disappeared and logging in to the dummy extension and downloading thousands of voicemails over ?period is onerous.

 

[AWS2P]

It only affected us starting yesterday for 2 customers, today for several more we discovered when they contacted us when none of their daily reports came in from the 3CX Server, so i think the block took some time to roll through all of 365's segments. Yesterday we sent out some welcome emails and they seemed to go thru fine, but one customer couldn't get reports, then today when it snowballed we dug deeper, and here we are.

Got something similar with reports
https://www.3cx.com/community/threads/report-emails-no-more-received-on-hosted-3cx.122129/

 

[SteveITS]

thousands of voicemails

I wonder if it's that volume that is working against you. I don't want to come across as obnoxious but I would think setting up a small VPS with one email account, once, would be better long term than downloading that volume of messages, occasionally. (with "thousands," a shared server mailbox may not allow/be happy sending that many)

tried using Gmail to relay the messages. When there was a brief service outage with Gmail guess what? Blacklisted

I don't follow this...Gmail blocking you sending? That might also be because of the volume. Did you use Workspace (https://support.google.com/a/answer/176600?hl=en), or a free Gmail account?

 

[Lee Cramman]

I wonder if it's that volume that is working against you. I don't want to come across as obnoxious but I would think setting up a small VPS with one email account, once, would be better long term than downloading that volume of messages, occasionally. (with "thousands," a shared server mailbox may not allow/be happy sending that many)


I don't follow this...Gmail blocking you sending? That might also be because of the volume. Did you use Workspace (https://support.google.com/a/answer/176600?hl=en), or a free Gmail account?

Not obnoxious at all - but inconvenience apart, our (small) organisation just doesn't have the skills / capacity to do so beyond myself (which introduces a single point of failure). Not to mention that, these days, it is not a trivial matter to set up and run smtp in a way that will be trusted and won't immediately be greylisted / blacklisted by the receiving server. Protecting against spam, DKIM, SPF, DMARC DNS... keeping an eye on real time black hole blacklists...

Unless you're talking about keeping 3CX smtp and then spinning up a dedicated mail server, in which case as soon as it has any outage it will be blacklisted just as any other service is.

We could use a service like Brevo alongside a fixed IP, but the monthly cost would be more than our 3CX hosting costs.

Sorry if the part about Gmail is confusing, what I was saying is that because (presumably) Google had a brief outage on their service, the 3CX smtp server blacklisted the Gmail address. Gmail itself did not block anything.

I guess the point I'm trying to make is that, logically, service outage (server temporarily unavailable) should not equal blacklisting. Only "user not found on this server" should. If an email address is blacklisted by 3CX smtp there should be a mechanism whereby we can tell / are informed and we should have a way of submitting addresses for removal from the 3CX blacklist or even a way of whitelisting them.