Microsoft 365 integration stopped working

[Graf]

Hi,
We have an on premise Debian installation of 3CX whose integration was working until recently.
Now it's not possible to activate or edit any of the options under "User sync" "Sign In" "Contact Sync" "Calendar Sync".
Now, every time i try to "Sign in to Microsoft 365" it tries to log in but no test status line in the management panel is displayed.

In 3cxSystemService.log i only get:

2023/03/10 14:46:10.234|26816|0038|Info|Office365AccessTokenManager.GetAsync START GetAcessTokenByCredentials
2023/03/10 14:46:10.383|26816|0030|Erro|Error while GetSubscriptionsAsync: The SSL connection could not be established, see inner exception.
2023/03/10 14:46:10.385|26816|0030|Info|Subscriptions:
ContactSubscriptions found: 0
SharedContactSubscriptions found: 0
EventSubscriptions found: 0
UnknownSubscriptions found: 0
UsersSubscription found: 0
Duplicate Subscriptions found: 0

I double checked the App Configuration in my Microsoft 365 Tenant, but everything seems to be as per Manual: https://www.3cx.com/docs/manual/microsoft-365/

3CX version is 18.0 (Build 908) with Professional Annual License

Please help, since all my phonebook relies on Shared Mailbox Sync

Many thanks

 

[SweetAction]

Are you accessing the 3CX pages using https? Is the ssl cert on that page valid?

 

[Graf]

Are you accessing the 3CX pages using https? Is the ssl cert on that page valid?

Yes, valid Let's Encrypt certificate on 3cx admin page (nginx)

Could it be that Microsoft 365 cannot verify Let's Encrypt generated certificates?

 

[SweetAction]

Yes, valid Let's Encrypt certificate on 3cx admin page (nginx)

Could it be that Microsoft 365 cannot verify Let's Encrypt generated certificates?

No LE certs should be fine.

I have no additional ideas without more logs or preferably a packet capture to see what's going on.

 

[Graf]

But shouldn't it use the cert / key pair generated in the Microsoft 365 Integration Page?
What does the webserver certificate has with Microsoft 365 App registration certificate to do?
Isn't the 365 Integration a different mechanism than the webserver certificate?

 

[SweetAction]

The webserver cert is used as part of the integration as callbacks happen to the webserver.

You can visit the login page of the client/mc without SSL (http on 80/5000) and notice the SSO button won't show.

 

[Graf]

Hi,
thank you in advance for your support.

Visiting the login page on port 5000 redirects me to the ssl part of the login (https) as part of the nginx default configuration:

if ($scheme = 'http'){
            set $block_remote "1";
        }
        if ($is_remote){
            set $block_remote "${block_remote}+1";
        }
        if ($block_remote = "1+1"){
            return 403;
        }

I could change it to verify what you are saying, but i think that it won't resolve my issue since i still have this error:

2023/03/13 10:02:50.387|491|0054|Info|Office365AccessTokenManager.GetAsync START GetAcessTokenByCredentials
2023/03/13 10:02:50.480|491|0054|Erro|Error while GetSubscriptionsAsync: The SSL connection could not be established, see inner exception.

Certificate (and chain) from Let's Encrypt is correct:


I still suspect that something has changed on M365 side since we didn't change anything on our side since v18 upgrade more than one year ago...
Do you know how and where i can check oauth2 authentication process?

 

[Graf]

I managed to resolve our issue.

Looks like 3cx app was using internal .NET store instead of using CAs from the ca-certificate package.

Deleted all .pfx certificates from /var/lib/3cxpbx/.dotnet/corefx/cryptography/x509stores/ca and Microsoft 365 sync started working again as expected.

 

[illcsales]

I managed to resolve our issue.

Looks like 3cx app was using internal .NET store instead of using CAs from the ca-certificate package.

Deleted all .pfx certificates from /var/lib/3cxpbx/.dotnet/corefx/cryptography/x509stores/ca and Microsoft 365 sync started working again as expected.

Great find @Graf
We also experienced this on one of our locations, odd that it did not affect all Office365 integrated sites.
Would be interested on why this happened in isolated cases and not all.

 

[SweetAction]

First - am glad to hear you solved it. I've not run into this issue, so thanks for documenting your steps and solution.

Second, regarding this:

Hi,
thank you in advance for your support.

Visiting the login page on port 5000 redirects me to the ssl part of the login (https) as part of the nginx default configuration:

You'll need to hit it from inside the network for 5000/80 to work. If your 3CX is hosted it's unlikely you will be able to do so.

 

[Graf]

Second, regarding this:

You'll need to hit it from inside the network for 5000/80 to work. If your 3CX is hosted it's unlikely you will be able to do so.

I was hitting from inside the local network, but as you can see from my previous post, nginx configuration is preventing from using non SSL connections triggering a 403 with HTTP -> HTTPS connection upgrade.

Must be a latest upgrade change, since i don't recall this behaviour to be implemented in previous versions (v16 and earlier).

Great find @Graf
We also experienced this on one of our locations, odd that it did not affect all Office365 integrated sites.
Would be interested on why this happened in isolated cases and not all.

I am interested as well, because even through VERBOSE logging the issue is not logged at all, thus preventing admins (me in this case) from effectively debugging the issue...

 

[SweetAction]

I was hitting from inside the local network, but as you can see from my previous post, nginx configuration is preventing from using non SSL connections triggering a 403 with HTTP -> HTTPS connection upgrade.

Must be a latest upgrade change, since i don't recall this behaviour to be implemented in previous versions (v16 and earlier).

I can confirm that at least on v18u6 that it's still possible to load both the web client and the management console over http (the management console will no longer display the sign in with microsoft button). You might have to use a browser that hasn't visited the page via https yet OR clear your HSTS cache (in chrome: chrome://net-internals/#hsts and then delete domain policy)

I'm aware of the nginx config, but that's not the reason it's loading over https each time - it's because of the hsts policy which gets set per browser on first visit - http is still in 3CX.

These are from the same install...

 

[JonnyM]

I'm having this same issue, trying to use the Sign-in with Microsoft button results in a redirect to webclient/signin/microsoft?jwt=true&ngsw-bypass and an HTTP 500 error.

The VM is running in Azure and was originally a PBX Express install from 2019, and is running Debian Buster if that helps to figure out why there are issues. There are two PFX files in here from 2020 - do you want a copy of them, do you want any log files, or shall I just remove them?

 

[weareit]

Hi - I had the same issue and this also resolved it for me. v18.0. Thank you for sharing the fix - I was pulling my hair out.

 

[jv5]

I managed to resolve our issue.

Deleted all .pfx certificates from /var/lib/3cxpbx/.dotnet/corefx/cryptography/x509stores/ca and Microsoft 365 sync started working again as expected.

Well I'll be...was having the same issue and this worked for us too. Also on v18.0

Fixed our <my.voip.server>/signin/microsoft HTTP 500 error too.

 

[Csut12]

We ran into seemingly same issue with Microsoft DynamicsCRM integration. The error was:

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

To fix, ran following commands:

mkdir ~/ca-old20230612
sudo mv /var/lib/3cxpbx/.dotnet/corefx/cryptography/x509stores/ca/*.pfx ~/ca-old20230612

Immediately after was able to re-Authorize the integration and test successfully.