My 3CX is working flawless part from that I have the same problem with email (notifications).
The problem is that authentification with the mail server don't work. It has nothing to do with ports, firewalls, IP addresses, size of email message, or such.
I also fail to send any kind of email notifycation if I try to authentificate by email server. Only as "Anonymous" towards mail server works (with no login or password in 3CX Phone System). I tried "all" combinations, all authentification methods, nothing works except as "Anonymous".
My setup:
Windows Server 2008 R2 (i64)
Exchange Server 2010 RC (i64)
3CX Phone System 7.1.7139
Mail don't route externaly. I have 3CX and Exchange on same server communicating with a "local" NIC, 192.168.0.1. Exchange uses the default receive connector on port 25. All IP addresses, IP4 and IP6 is accepted. I tried Basic (with and without TLS), Integrated Windows, and Exchange Server authentification. I tried those in all combos with all permision groups (Exchange Users, Exchange Servers, etc). No luck. Only setting that matters to get email notifications going is "Permision Group" Anonymous - which of cource don't use any authentification at all, meaning in this setup that my Exchange Server accept email (for relaying) from any computer without any kind of password ... on port 25 ... luckily for me my ISP blocks port 25, so no spammer can use my Exchange Server from outside for relaying. But of cource authentification should work with 3CX so that I could have authentification on for security. Even if only basic authentifiaction.
Speaking of ... can you change the SMTP port in 3CX?
PS. Don't mind that I'm running Windows Server 2008 R2 and Excahnge Server 2010. I had the same problem with Windows Server 2008 and Exchange Server 2007.