Office365 IMAP/POP basic auth retirement October 2020

Question

Office365 IMAP/POP basic auth retirement October 2020

Closed this issue 5 years ago · 31 comments

Microsoft have announced the retirement of basic auth for IMAP/POP in Office365 in the following blog post:

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-auth-and-exchange-online-february-2020-update/ba-p/1191282

It is possibly to early to comment but will there be an intention to implement modern auth, if it's even possible, in the IMAP/POP client within Mailkit?

Answer 1 · 2020-02-28T16:34:40.000Z

They're replacing basic auth with OAuth2 which MailKit already supports. You will, however, need to use a 3rd party library to get an authentication token to use with MailKit as MailKit does not provide such an API.

Once you have the token, you would authenticate like this:

client.Authenticate (new SaslMechanismOAuth2 ("username@outlook.com", oauth2_token));

Answer 2 · 2020-04-15T12:39:20.000Z

Has anyone actually managed to get this working? I've got to the point where I can get the authentication token from Office365, but the Authenticate call fails with:

HResult=0x80131500
Message=Authentication failed.

This is what I have so far, using MSAL:

var scopes = new[] { "https://graph.microsoft.com/.default" };
var app = PublicClientApplicationBuilder.Create(ClientId).WithAuthority(AadAuthorityAudience.AzureAdMultipleOrgs).Build();
var tokenResult = await app.AcquireTokenByUsernamePassword(scopes, username, password).ExecuteAsync(cancellationToken);

using var client = new ImapClient();
await client.ConnectAsync("outlook.office365.com", 993, SecureSocketOptions.Auto, cancellationToken);
await client.AuthenticateAsync(new SaslMechanismOAuth2(username, tokenResult.AccessToken), cancellationToken);

Same error if I change the scope to https://outlook.office365.com/.default - the access token is returned, but the Authenticate call fails.

Answer 3 · 2020-04-15T18:49:30.000Z

@RichardD2 this may be useful to you? https://stackoverflow.com/questions/43473858/connect-to-outlook-office-365-imap-using-oauth2/60773366#60773366

Answer 4 · 2020-04-15T19:08:44.000Z

@jstedfast Unfortunately not.

I've tried granting the permissions under Microsoft Graph -> IMAP and Microsoft Graph -> Mail:

but I still get the "authentication failed" exception, regardless of whether I use https://graph.microsoft.com/IMAP.AccessAsUser.All or the recommended https://graph.microsoft.com/.default as the scope.

I'm assuming that MailKit's OAuth2 authentication works with other providers, so it's either a configuration error or a bug in Microsoft's implementation.

Answer 5 · 2020-04-15T19:21:01.000Z

Yea, the OAuth2 SASL mechanism is dead simple (it just adds the username and access_token to a blob and base64 encodes it, essentially). It also works with Google, so I know it's fine.

At least 1 other person has asked me about how to get oauth2 working with hotmail/office365, so you aren't alone.

StackOverflow is full of people asking how to make it work with javamail, python, php, etc.

Answer 6 · 2020-04-15T19:23:31.000Z

@jstedfast Thanks - glad to know it's not just me!

Answer 7 · 2020-04-16T12:45:15.000Z

@RichardD2 I've contacted the Office365 team and they have asked me to connect you with them so they can help you figure out what is going wrong. Can you email me at jestedfa@microsoft.com?

Answer 8 · 2020-04-18T02:51:32.000Z

@RichardD2 give this a try: https://outlook.office365.com/IMAP.AccessAsUser.All

Answer 9 · 2020-04-20T09:52:19.000Z

@paulflo150 That worked, thanks. 👍

Now I just need to work out how to access a shared mailbox using OAuth2. The usual trick of appending \shared-mailbox-alias to the username results in the same "authentication failed" error.

Connecting without specifying the shared mailbox alias and checking the SharedNamespaces collection as suggested in #965 doesn't help, because that collection is empty.

Answer 10 · 2020-04-20T14:09:19.000Z

@RichardD2 since you've got it kinda-mostly working, could you post the code you are using to get the auth_token? I've been getting questions from other people about this as well, and. haven' had the time to investigate how to do this using Microsoft Graph.

Thanks!

Answer 11 · 2020-04-20T14:34:05.000Z

@jstedfast No problem.

At the moment, I'm just using the username/password flow with the MSAL library:

const string ClientId = "..."; // The "Application (client) ID" from the Azure apps portal
const string UserName = "..."; // The Office365 account username - user@domain.tld
SecureString Password = ...;   // The Office365 account password

// Get an OAuth token:
var scopes = new[] { "https://outlook.office365.com/IMAP.AccessAsUser.All" };
var app = PublicClientApplicationBuilder.Create(ClientId).WithAuthority(AadAuthorityAudience.AzureAdMultipleOrgs).Build();
var authenticationResult = await app.AcquireTokenByUsernamePassword(scopes, UserName, Password).ExecuteAsync(cancellationToken);

// Authenticate the IMAP client:
using var client = new ImapClient();
await client.ConnectAsync("outlook.office365.com", 993, SecureSocketOptions.Auto, cancellationToken);
await client.AuthenticateAsync(new SaslMechanismOAuth2(UserName, authenticationResult.AccessToken), cancellationToken);

// Check that we can read the messages in the inbox:
await client.Inbox.OpenAsync(FolderAccess.ReadOnly, cancellationToken);
var messages = await mailFolder.FetchAsync(0, -1, MessageSummaryItems.UniqueId | MessageSummaryItems.InternalDate, cancellationToken);
foreach (var item in messages)
{
    Console.WriteLine("{0}: {1}", item.UniqueId, item.InternalDate);
}

I'm using the same credentials to obtain the token that I was using with the "basic" authentication option.

I followed the MS docs to register the application:
Quickstart: Register apps with Microsoft identity platform | Microsoft Docs

Under Authentication, I selected "Accounts in any organisational directory", and "Treat application as a public client" is set to "Yes".

On the API Permissions page, I selected "Add a permission", then Microsoft Graph => Delegated Permissions => IMAP => IMAP.AccessAsUser.All; then clicked "Grant admin consent for [org]".

Answer 12 · 2020-04-20T14:38:23.000Z

Awesome, thanks @RichardD2 !

Answer 13 · 2020-04-28T09:42:26.000Z

With help from Sivaprakash Saripalli at Microsoft, I've managed to update the code to access a shared mailbox as well. It's as simple as passing the email address of the shared mailbox instead of the UserName to the SaslMechanismOAuth2 constructor.

const string ClientId = "..."; // The "Application (client) ID" from the Azure apps portal
const string UserName = "..."; // The Office365 account username - user@domain.tld
SecureString Password = ...;   // The Office365 account password
const string Mailbox = "..."; // The shared mailbox name - mailbox@domain.tld

// Get an OAuth token:
var scopes = new[] { "https://outlook.office365.com/IMAP.AccessAsUser.All" };
var app = PublicClientApplicationBuilder.Create(ClientId).WithAuthority(AadAuthorityAudience.AzureAdMultipleOrgs).Build();
var authenticationResult = await app.AcquireTokenByUsernamePassword(scopes, UserName, Password).ExecuteAsync(cancellationToken);

// Authenticate the IMAP client:
using var client = new ImapClient();
await client.ConnectAsync("outlook.office365.com", 993, SecureSocketOptions.Auto, cancellationToken);
await client.AuthenticateAsync(new SaslMechanismOAuth2(Mailbox, authenticationResult.AccessToken), cancellationToken);

// Check that we can read the messages in the inbox:
await client.Inbox.OpenAsync(FolderAccess.ReadOnly, cancellationToken);
var messages = await mailFolder.FetchAsync(0, -1, MessageSummaryItems.UniqueId | MessageSummaryItems.InternalDate, cancellationToken);
foreach (var item in messages)
{
    Console.WriteLine("{0}: {1}", item.UniqueId, item.InternalDate);
}

Answer 14 · 2020-04-29T22:08:58.000Z

Thanks for sharing that.

Unrelated to MailKit, but personal mail like hotmail for instance isn't supported by MSAL right?

Answer 15 · 2020-04-30T01:29:03.000Z

@paulflo150 I believe it is, actually, but don't quote me on that.

Answer 16 · 2020-05-12T17:09:07.000Z

@jstedfast, If you do manage to get this working with Graph, please let me know. I've been struggling to accomplish this too with the mix of information that's out there.

Thanks!

Answer 17 · 2020-07-11T18:35:50.000Z

@carmanj You may find this helpful: https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md

Answer 18 · 2020-09-17T22:51:43.000Z

@carmanj You may find this helpful: https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md

Thanks for the pointer! Is there any way to get that example working in ASP.NET with VB? I keep getting a "ActiveX control '8856f961-340a-11d0-a96b-00c04fd705a2' cannot be instantiated because the current thread is not in a single-threaded apartment." error.

Thanks!

Answer 19 · 2020-09-18T09:13:36.000Z

@carmanj The CLSID 8856f961-340a-11d0-a96b-00c04fd705a2 is associated with the WebBrowser control. It looks like AcquireTokenInteractive is meant for use in a desktop application, and is trying to open an interactive form on the server, which isn't what you want.

For ASP.NET, you'll need to use a different flow:
Build a web app that calls web APIs - Microsoft identity platform | Microsoft Docs

Answer 20 · 2020-09-21T18:30:25.000Z

@carmanj The CLSID 8856f961-340a-11d0-a96b-00c04fd705a2 is associated with the WebBrowser control. It looks like AcquireTokenInteractive is meant for use in a desktop application, and is trying to open an interactive form on the server, which isn't what you want.

For ASP.NET, you'll need to use a different flow:
Build a web app that calls web APIs - Microsoft identity platform | Microsoft Docs

Yeah, as I got looking through the exception afterward, I was wondering if that was the issue.

Our app is somewhat hybrid in nature. Has VB ASP.NET front end to manage account configurations and a C# based windows service, which uses IMAP to fetch emails and process their contents to link to records as part of a longer-running process that runs every 15 minutes. I've tried using Microsoft's VanillaJS example to run through the authorization popups and get a token back. The idea would then be to persist it in the database, so the service can go ahead and log in to the box on its schedule.

If I grab the token that the frontend gets and manually run the service with it, I keep getting "S: B00000001 NO AUTHENTICATE failed." in the logging for MailKit. There's a line previous to that "C: B00000001 AUTHENTICATE XOAUTH2 " with a base64 encoded string appended to the end in the format of user=user@domain.com�auth=Bearer xxx��. The bearer token contains a JWT which appears to also be base64 encoded. Any thoughts on why that is coming back failing to authenticate? Do I actually need to base64 decode the JWT that gets passed in as a token to the New MailKit.Security.SaslMechanismOAuth2() constructor?

Thanks to both of you, @jstedfast and @RichardD2 for your help!

Answer 21 · 2021-01-13T14:53:29.000Z

Hi!

So I'm having literally the same issue as RichardD2 has with the main difference that I am not using .NET Core

I successfully retrieve the token, but Authentication with Sasl keeps failing. Any idea why?

My code is literally this:

using System;
using System.Threading;
using System.Threading.Tasks;
using MailKit.Net.Imap;
using MailKit.Security;
using Microsoft.Identity.Client;

namespace OAuth2ExchangeExample
{
	class Program
	{
		const string ExchangeAccount = "My_Exchange@Email.com";

		public static void Main(string[] args)
		{
			using (var client = new ImapClient())
			{
				try
				{
					client.Connect("outlook.office365.com", 993, SecureSocketOptions.SslOnConnect);

					if (client.AuthenticationMechanisms.Contains("OAUTHBEARER") ||
						client.AuthenticationMechanisms.Contains("XOAUTH2"))
					{
						AuthenticateAsync(client).GetAwaiter().GetResult();
					}

					client.Disconnect(true);
				}
				catch (Exception ex)
				{
					Console.WriteLine(ex.Message);
					Console.ReadKey();
				}
			}
		}

		static async Task AuthenticateAsync(ImapClient client)
		{
			var options = new PublicClientApplicationOptions
			{
				ClientId = "ClientId",
				TenantId = "TenantId",
				RedirectUri = "https://login.microsoftonline.com/common/oauth2/nativeclient"
			};

			var scopes = new string[] {
				"https://graph.microsoft.com/.default"
			};

			var cancellationToken = new CancellationToken();

			var app = ConfidentialClientApplicationBuilder.Create("ClientId")
				.WithClientSecret("ClientSecret")
				.WithAuthority(new Uri("https://login.microsoftonline.com/" + options.TenantId + "/"))
				.Build();

			var authenticationResult = app.AcquireTokenForClient(scopes)
				.ExecuteAsync(cancellationToken);

			var token = authenticationResult.Result;

			// Note: We use authToken.Account.Username here instead of ExchangeAccount because the user *may* have chosen a
			// different Microsoft Exchange account when presented with the browser window during the authentication process.
			SaslMechanism oauth2;

			if (client.AuthenticationMechanisms.Contains("OAUTHBEARER"))
				oauth2 = new SaslMechanismOAuthBearer(ExchangeAccount, token.AccessToken);
			else
				oauth2 = new SaslMechanismOAuth2(ExchangeAccount, token.AccessToken);

			client.Authenticate(oauth2);
		}
	}
}

Answer 22 · 2021-01-13T18:09:19.000Z

@Jedrzej94 I'm not sure if you've seen https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md or not - but it may be helpful (although it seems like if you want to use OAUTH2 as a service rather than as a client, then a different approach is needed based on #1126).

I've only ever gotten OAUTH2 to work using the Microsoft.Identity API rather than the Microsoft.Graph API, so unfortunately I don't currently have an answer to issues arising from using the Microsoft.Graph API :(

Answer 23 · 2021-01-13T19:43:24.000Z

@jstedfast , for some obnoxious reason the only scope that returns token for me is: "https://graph.microsoft.com/.default"

with that link:
https://github.com/jstedfast/MailKit/blob/master/ExchangeOAuth2.md

I managed to find out that my URL Reply needed some adjusting (because Interactive method of that code informed me about this). So I have configured it now in AAD > Authentication > Reply URL as: "https://login.microsoftonline.com/common/oauth2/nativeclient"

That lead to:

A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.
Trace ID: ccd5bc23-fa86-4a3a-86c3-6b745a0f0200
Correlation ID: d3bf1bab-d54b-485f-b1de-8c1e83ccba0d
Timestamp: 2021-01-13 19:38:57Z

So I found out that I need to enable Public client flows in AAD > Authentication and so I did but I still receive an error.

This is the code I'd rather use to avoid "Interaction" in ASP.NET (it works only for [...]/.default scope)

var confidentialClientApplication = ConfidentialClientApplicationBuilder
					.Create(Options.ClientId)
					.WithClientSecret(ClientSecret)
					.WithAuthority(new Uri("https://login.microsoftonline.com/" + Options.TenantId + "/"))
					.Build();

var authenticationResult = confidentialClientApplication.AcquireTokenForClient(Scopes)
								.ExecuteAsync(cancellationToken);

var authToken = authenticationResult;
var oauth2 = new SaslMechanismOAuth2(ExchangeAccount, authToken.Result.AccessToken);

edit://
About your 2nd link. I tried the scopes you recommend over all and:

AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope email offline_access https://outlook.office.com/SMTP.Send is not valid.
Trace ID: ccead844-9c77-4b93-aabd-d700ac6b2100
Correlation ID: 15347971-0975-4b8c-9408-9b081006b998
Timestamp: 2021-01-13 19:44:42Z

Answer 24 · 2021-01-13T22:52:05.000Z

This may be helpful reading: https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-net-instantiate-confidential-client-config-options

Answer 25 · 2021-01-13T23:02:11.000Z

This documentation explains the difference between "Public Client Applications" and "Confidential Client Applications" and you should all make sure you are using the correct one.

The link in my previous comment explains how to get the needed values for a Confidential Client Application.

https://docs.microsoft.com/en-us/azure/active-directory/develop/msal-net-instantiate-public-client-config-options explains how to obtain the needed info for a Public Client Application.

Answer 26 · 2021-01-14T16:56:10.000Z

I followed what RichardD2 did for the shared mailbox, but I'm getting this error in the logs
S: A00000002 OK CAPABILITY completed.
C: A00000003 NAMESPACE
S: A00000003 BAD User is authenticated but not connected.
C: A00000004 LIST "" "INBOX"
S: A00000004 BAD User is authenticated but not connected.
S: * BYE Connection closed. 14

Not sure what I did wrong. Pop3Client works by the way, but the IMapClient has the features I need :|

Answer 27 · 2021-01-14T16:59:25.000Z

@azalpacir Are you trying to open a mailbox which the user doesn't have permission to view? I've seen exactly the same log messages when that happens, and the Inbox property returns null. [#1135]

Answer 28 · 2021-01-14T18:23:02.000Z

@azalpacir @RichardD2 just submitted a bug about that in the last day or so.

Answer 29 · 2021-01-14T21:09:15.000Z

EDIT: Disregard. IT had disabled IMAP and POP access.

Seem to be having a similar (but not exact) issue to everyone here.
I've been successful in retrieving tokens using PublicClientApplicationBuilder (both AcquireTokenInteractive and AcquireTokenByUsernamePassword) but I get an "Authentication Failed" message when I call:
await client.AuthenticateAsync(oauth2);

Logs look like this:

Connected to imaps://outlook.office365.com:993/
S: * OK The Microsoft Exchange IMAP4 service is ready.
C: A00000000 CAPABILITY
S: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
S: A00000000 OK CAPABILITY completed.
C: A00000001 AUTHENTICATE XOAUTH2 dXN...
S: A00000001 NO AUTHENTICATE failed.

Hoping that someone has gotten farther than I have here.

Answer 30 · 2021-01-14T21:15:20.000Z

@azalpacir Are you trying to open a mailbox which the user doesn't have permission to view? I've seen exactly the same log messages when that happens, and the Inbox property returns null. [#1135]

Pop3Client works as I mentioned. I also tried switching from the main email account to the shared one after logging in the browser and I can read the messages.

When you say it doesn't have access what specific access rights are we looking at. So I can relay it to our service desk and update the necessary access. Thanks.

Answer 31 · 2021-03-04T07:08:37.000Z

Was able to make it work with the shared mailbox by using the username of the shared mailbox instead of the domain email name of shared mailbox.