It's one of the reason to rewrite mail system in 4 and extend clearly PHPMailer.
https://github.com/LimeSurvey/LimeSurvey/blob/3ca1be07878d3e3953a7aa38bf7fb5f85345ccb8/application/core/LimeMailer.php#L495
Sample plugin : https://github.com/LimeSurvey/LimeSurvey/tree/develop/application/core/plugins/mailSenderToFrom (more easy this one)

According to this http://www.pmail.com/newsflash.htm Google will require Oauth2 by the end of May 2022 - this month. Would it be worth having this issue bumped up the priority list?

The cost of the assessment typically varies between $10,000 -$75,000 (or more) depending on the size and complexity of the application; smaller applications may see costs at a lower threshold of $4,500.

leave google …

And again : it can be done in plugin

"...leave Google"

Lol...

"...can be done in plugin": Most certainly but many of the existing features could be plugins, including all the SMTP settings. I think a more nuanced rule is used in practice ;)

image.png (9,169 bytes)   

image.png (9,169 bytes)   

Did you really read the link

"Every app that requests access to restricted scope Google user´s data … may see costs at a lower threshold of $4,500."

I confirm : leave google.

Irrelevant to this issue (relates to Android app publishing)?

LimeSurvey doesn't seem to be leaving Google - see image for three Google specific settings in Global Configuration.

NB Oauth2 isn't even just for Google, it's a standard (rammed through by Google) for what that's worth

So according the schedule Google has deactivated LSA a long time ago.
I did not hear from our user base so far that sending invitations using Gmail would not work.
So, it is not possible to use Google Mail with LimeSurvey now?

I think they planned to do it 2020 but delayed it due to the pandemic.

Seems that it the switchover date for mandatory Oauth2 is now 30 May 2022:

https://support.google.com/mail/thread/153515128/which-email-client-will-work-with-gmail-after-30th-may-2022?hl=en
https://github.com/k9mail/k-9/issues/5956
https://www.ghacks.net/2022/04/06/thunderbird-91-8-0-makes-important-changes-to-google-mail-accounts/

Found the actual announcement on LSAs:

https://support.google.com/accounts/answer/6010255

Please find below some comments about the analysis of the ticket

About the protocol

https://stackoverflow.com/questions/65228456/how-does-phpmailer-use-oauth/65230616#65230616

What's difficult about XOAUTH2 is obtaining that access token. To get hold of it requires that you run an HTTP service (on top of whatever other services you are running) and provide it with the correct ID and secret, which you submit and receive an access token in response, which you then use in your XOAUTH2 request. PHPMailer provides a script that helps you to do this for Google, Microsoft and Yahoo services, and others via the numerous adapters that are available for the PHPLeague's OAuth2 client package.

IMHO, OAuth2's monumentally confusing structure, ambiguity, and complexity creates excessive overhead and many opportunities for error, and it's all ultimately secured by a username and password, all of which undermines the security of the whole exercise. What doesn't help is that there is no standardisation, and the procedures you must follow to obtain tokens can be completely different across providers (hence the need for different adapters), and is subject to frequent change, making it hard to document. However, since Google imposes it on everyone, you have no choice but to go along with it.

Analysis

Please find below some comments about the analysis done:

Library

For each OAuth2 provider a specific library is needed

Data

In general, I understand that to send the mail it will be needed a client_id, a client_secret, and the Refresh Token.
In the case of Azure you also need a tenantId.
I don't know how it will be with other providers.

Endpoint for getting the Refresh Token

The Refresh Token is assigned by the provider.
An endpoint is required to receive it from the provider and save it.

• It has to be HTTPS
• The URL is configured in the Google console (or the provider setting).

UI for Refresh Token

Global Settings > Email Settings should be updated

• It should have some warning that OAUTH only works if HTTPS is enabled.
• It should ask you:
  • Provider (Google, Microsoft, Yahoo, etc.)
  • Client ID
  • Client Secret
    When saving (or maybe with a new get refresh token button), it should send you to the provider page to confirm the permissions. The new endpoint would receive the message and save the refresh token in a global setting (no UI).

We could also fetch the settings with a plugin event.

Plugin event

Ideally there would be new Plugin Events added so that plugins can register new providers.
The provider is used in two moments:

When fetching the refresh token

We could implement the UI for refresh token as plugin settings.
This common behaviour could be on a BaseSMTPOauth class.

When sending the mail.

To get the AbstractProvider instance that PHPMailer needs, additional parameters may be needed (such as the tenantId in the case of Azure), so somehow the plugin would have to indicate what additional fields should be add in Global Settings.

Also, we could set the provider on a plugin event (ex: beforeEmail)

Don't try to develop OAuth system,
PHPMailer already have solution : https://github.com/LimeSurvey/LimeSurvey/blob/master/application/third_party/phpmailer/src/OAuth.php

The doc is here …
https://github.com/PHPMailer/PHPMailer/wiki/Gmail-XOAUTH2-Using-Google-API-Client

I really think it must be a Plugin , core plugin yes, but a plugin.

Don't try to develop OAuth system,

Never intended to

Never intended to

Seems need specific library for GMail

composer require google/apiclient ^2.11

Then really better in a plugin

I would suggest that as a first step you add OAUTH support for Google as described in the PHPMailer docs.
After that is implemented we will see how we go from there.

I would suggest that as a first step you add OAUTH support for Google as described in the PHPMailer docs.

You mean a PoC?
Or adding that to Lime with all the configuration screens?

If you intended the latter, I guess you are OK with the analysis and approach, right?

Or adding that to Lime with all the configuration screens?

A lot of configuration screen + a complete library

A good reason to set it as Plugin ?
No ?
@c_schmitz ?

Picking this up. As a plugin :)

Hi All,

Let me show you how about the progress on this story.

We have developed a core plug-in that:

1- Pickups the credentials for connecting to Google services.

2 - Calls Google services to get a refresh-token, that will be used when sending emails.

3 - Intervene during the email sending process to avoid sending the emails through the regular means and send them directly through Google services.

Please find below a few screenshot.

Working on this proof-of-concept we have come with the following issues or observations.

Https required

Google requires an https URL callback when setting up the service. So HTTPs will be required for using this plugin. We will be adding a check on that as well as a warning if not having the proper schema.

From Address

Google seems to overwrite the from address and use the one related to the account.

New Email Events

Right now we only have beforeXxxEmail event, which is used generally for decorating the email (Ex. adding a header), or for sending it through a different non-standard mean.

As the plug-ins don't have a synchronization mechanism, if there are two plug-ins using the same beforeXxxEmail event, is hard to know for sure which plug-in will run first and which will run later.

Because of that we believe we need a specific event named beforeSendingEmail. This way we will be able to separate two different processes:

A - Decorating the email (as adding a header or setting specific form addresses depending on the survey)

B - Sending it (as sending it through SMTP, OAuth, or even other gateways).

Global setting screen

We will need to inform the administrators that the OAuth plugin is active. If not administrator could get confused and think that the emails will be sent through regular means.

Right now for this proof-of-concept we are going to be doing it through JavaScript on an event. We believe we may need something a little bit more robust as there could be many other email plug-in.

We don't have a clear view on this but we picture having an email plugin coordinator, which can query the different email plug in and, for example, identify if the emails will be sent to a regular mean or through a plug-in email.

The global settings screen could use this new blocking email coordinator for knowing if the SMTP credential fields should be disabled or not, and also for showing a text message that says which plug-in will be handling emails.

Porting code to new plugins

Looking to the Future the current meaning for sending emails could be relocated in one or many core plug ins.

As the plug-ins don't have a synchronization mechanism, if there are two plug-ins using the same beforeEmail event, is hard to know for sure which plug-in will run first and which will run later.
Of that we believe we need a specific event named beforeSendingEmail. This way we will be able to separate two different processes:

beforeEmail receive whole LimeMailer (extend PHPMailer) as ->get('mailer'), then it can be used for anything that is related to mail in my opinon.

You can use priority in config.xml to be sure to use this way to send email after all olther are done. It's done for this.

But : we always have incompatibility issue with other plugin.
For example : you can not use https://github.com/Xites/LS_mailDKIM with google.

Or : developer can not really be sure other plugin broke somthing : https://gitlab.com/SondagesPro/mailing/noMailToExample must happen before google here for example.

POC: https://github.com/LimeSurvey/LimeSurvey/tree/feature/15664-Google-OAuth-plugin

Great !

Maybe PR as draft ? Hard to compare here

PoC: https://github.com/LimeSurvey/LimeSurvey/pull/2868

Just to add to this feature request, the issue also now affects those who use Microsoft services for their emails (i.e. huge numbers of public and private sector organisaitons) because basic authentication has also been dropped by Exchange online at the beginning of this year:

https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online

Hi All,

Picking up a bit on this.

beforeEmail receive whole LimeMailer (extend PHPMailer) as ->get('mailer'), then it can be used for anything that is related to mail in my opinon.

So, you say that we should extend LimeMailer for sending emails through google, right?
If that's the case, how would admins choose what Mailer to use?
Config file?

If that's the case, how would admins choose what Mailer to use?

I don't understand ?
Plugin choose what to do.

Plugin choose what to do.

Plugin can choose what mailer to use?

https://github.com/LimeSurvey/LimeSurvey/blob/c1b544a03d05ba7302a3457c518c4722600c3c59/application/core/LimeMailer.php#L508

Plugin get whole LimeMailer then can extend it if needed .

But right : must be PHPMailer compatible.

You can use priority in config.xml to be sure to use this way to send email after all olther are done. It's done for this.

Priority is hard to coordinate among plugin developers.
As an offtopic, we could have something like "runBefore: PluginName".

I don't like the priority thing a lot.

I don't feel it was designed for cases like this, where you have 2 whole different processes: Sending Emails and Decorating them.
Yes, it could work, but I feel it is fragile and it was not designed for that.

Also, on the other side, if there are multiple plugins for sending emails, why all those should react to an event if we already know which one should do the work already.

You already mention about incompatibility or how one plugin can mess other plugins.

It is like we are talking not only about priority (sequence) but also coordination among plugins (sequence + general flow decisions like knowing which plugins should run).

Plugin get whole LimeMailer then can extend it if needed .

Yes, but if the mailer is replaced inside the plugin, how does the process continue when the event continues on the mailer ?
The situation you say would be like this:
1 - Mailer dispatch events
2 - Plugin replaces mailer (notice there is no output parameter on the event for returning a mailer)
3 - Mailer continues after plugin ? How? Which Mailer? Original one I guess.

Right now we only have beforeXxxEmail event, which is used generally for decorating the email (Ex. adding a header), or for sending it through a different non-standard mean.
As the plug-ins don't have a synchronization mechanism, if there are two plug-ins using the same beforeXxxEmail event, is hard to know for sure which plug-in will run first and which will run later.

My bad. I was confused and mixing subjects.
I believe we should have a separate event for sending emails, but it is not needed now for the solution.

OAuth plugin just need to set settings on mailer.

Please, for now, let's disregard the whole priority + sequence issues. Not an issue I believe.
Will keep you updated.

Sorry about all the back and forth, actual needs and code-reorgaization-desires got mixed. :)
So, after reviewing, this is what is pending to be done.

Thoughts?

New Mail Method

There should be a new mail method available from global settings: SMTP with OAuth.

Provider Registration - new Event listSMTPOauthPlugins

Limesurvey needs to know which providers are available so they are listed to the admin.
The admin can choose the provider from the Global Settings.

Ideally there would be new Plugin Events added so that plugins can register new providers (similar to export writers).
The providers would be shown on a dropdown.

There would be a new event for that: listSMTPOAuthPlugins.

After Selecting Provider: Informing Configuration Requirements - new Event afterSelectSMTPOAuthPlugin

The OAuth mechanism is based on a refresh token, that the client uses to authenticate to the provider.
This should be triggered by the admin. The admin should be invited to run the process with a message, afer setting the smtp provider.

The message would come from the plugin itself from a new event: afterSelectSMTPOAuthPlugin

Specific Configuration Parameters

Probably each OAuth mechanism would need some own parameters/settings to get it, and hence configuration screen.

We could implement the UI for refresh token as plugin settings.
If need custom views, they could be created inside the plugin.

Some of the common behaviour identified could be on a new BaseSMTPOauth plugin class.

Configuration Stage

The admin needs to follow the instructions from the plugin and configure the refresh token.
This could happen from the plugin settings.

Some of the common behaviour identified could be on a new BaseSMTPOauth plugin class.

Sending Email Stage - new Event newSMTPOAuthConfiguration

PHPMailer needs a provider (class which extends AbstractProvider) to authenticate with the provider.
The provider needs to be set from the plugin.

We would use a new event: newSMTPOAuthConfiguration

From LimeMailer (https://github.com/LimeSurvey/LimeSurvey/blob/c1b544a03d05ba7302a3457c518c4722600c3c59/application/core/LimeMailer.php#L156), a new case branch would be added to trigger an event for a specific plugin (usage of target parameter when dispatching the event).

3 - Mailer continues after plugin ? How? Which Mailer? Original one I guess.

It's same mailer, just partially resetted : if youupdate at start of process : it's not updated (before https://github.com/LimeSurvey/LimeSurvey/blob/c1b544a03d05ba7302a3457c518c4722600c3c59/application/core/LimeMailer.php#L233 init again)
I a plugin update mialer, it update for all action after using same instance.https://github.com/LimeSurvey/LimeSurvey/blob/c1b544a03d05ba7302a3457c518c4722600c3c59/application/core/LimeMailer.php#L224
For default instance : https://github.com/LimeSurvey/LimeSurvey/blob/c1b544a03d05ba7302a3457c518c4722600c3c59/application/core/LimeMailer.php#L200 init happen.

As an offtopic, we could have something like "runBefore: PluginName".
I don't like the priority thing a lot.
I don't feel it was designed for cases like this, where you have 2 whole different processes: Sending Emails and Decorating them.
Yes, it could work, but I feel it is fragile and it was not designed for that.

The original idea about priority was a complete "PluginA use PluginB" "PluginA need PluginB" etc …

But priority do the trick, if you have a "decorative" plugin : just remind to set priority up to 0.

Yes : plugin can not be compatible with ALL plugins … only according to specific plugins …

3 - Mailer continues after plugin ? How? Which Mailer? Original one I guess.

Just please disregard. That part of the conversiton got off-track. :)
I got confused a bit and probably drove the conversation crazy

But this part seems totally related :)

If you don't reset instance, but set to another mailer https://github.com/LimeSurvey/LimeSurvey/blob/34d67e356dd997fa30c125c4dbbde9b188866b11/vendor/phpmailer/phpmailer/src/PHPMailer.php#L215
(for example POP3)
The same mailer was used after :)

yes : if you use POP3 , core config are not really used …

If you don't reset instance, but set to another mailer

I got lost. If you feel it has to be considered for the new developemtn, and only then, can you please pickup and elaborate more?
Thanks!

I got lost. If you feel it has to be considered for the new developemtn, and only then, can you please pickup and elaborate more?

I don't understand : what you done here is OK : https://github.com/LimeSurvey/LimeSurvey/pull/2868/files#diff-faf63dfcf1cb20a354eedda8d9213e6924846a8176559153c8b789f79d18b6f6R425

I just add

if ($limeMailer->AuthType == 'XOAUTH2') {
    return;
}

because we don't need to call it again

And maybe time too ? We need to test if token still OK ?

what you done here is OK : https://github.com/LimeSurvey/LimeSurvey/pull/2868/files#diff-faf63dfcf1cb20a354eedda8d9213e6924846a8176559153c8b789f79d18b6f6R425

Agree

As to get aligned... These are the next steps I am thinking for this story.
https://bugs.limesurvey.org/view.php?id=15664#c73911

Any comment about that? About the new events?

Any comment about that? About the new events?

I alredy add own plugin event inside my plugins, one plugin is done only for this (replace beforeEmail event)
https://gitlab.com/SondagesPro/mailing/eventBeforeTokenEmailExtended/-/blob/master/eventBeforeTokenEmailExtended.php#L50

About updating Global settings : @ollehar didn't work on a new solution for plugins ?

I alredy add own plugin event inside my plugins

These would be new generally available plugin events called from outside the plugin.

About updating Global settings ?

The JS workaround we did on the PoC would go away. That was just for the PoC.
We would use the approach described couple of posts above (https://bugs.limesurvey.org/view.php?id=15664#c73911)

ollehar didn't work on a new solution for plugins

We wouldn't use that as the new screens would be inside the plugin

We would use the approach described couple of posts above (https://bugs.limesurvey.org/view.php?id=15664#c73911)

There should be a new mail method available from global settings: SMTP with OAuth.

Then you add it in core, not in core plugins ?

Then you add it in core, not in core plugins ?

We would be updating core to support a new email sending method: OAuth SMTP.
The sending would be done through core plugins.

We would be updating core to support a new email sending method: OAuth SMTP.

Here : it's to add it in Global settings, right ?

Maybe like export ? List export ? https://manual.limesurvey.org/Export_plugin_development#listExportOptions

But still lack of the settings in global settings (we really need a global event to add input on existing form …)

We would be updating core to support a new email sending method: OAuth SMTP.

Here : it's to add it in Global settings, right ?

Yes

Maybe like export ? List export ? https://manual.limesurvey.org/Export_plugin_development#listExportOptions

https://bugs.limesurvey.org/view.php?id=15664#c73911
Provider Registration - new Event listSMTPOauthPlugins

Provider Registration - new Event listSMTPOauthPlugins

It's the point, why not Provider Registration - new Event listEmailSendPlugins

Provider Registration - new Event listSMTPOauthPlugins

It's the point, why not Provider Registration - new Event listEmailSendPlugins

Could be, not sure if the same event can be used for all.
My idea: Just start with my suggestion and then if we see the event applies in more general terms, we just rename them.

PR: https://github.com/LimeSurvey/LimeSurvey/pull/3054

These are a couple of core plugins to send mails through google and azure.

Testing Instructions:
1 - Activate a plugin.
2 - Set the settings and Configure it (get refresh token from provider)
3 - Selectthe plugin on the global sttings as to send emails with the plugin
4 - Send emails (ex: using tokens)

Please find below more context on how to setup the environment for Azure.

• General OAuth Azure Process
  https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

• Env Setup
  https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app

Similar for Google:
https://support.google.com/cloud/answer/6158849?hl=en

@Indispirit can you help testing maybe?

Just to add to the comment above, I have tried to test this, but I am encountering issues with both Azure and Google OAuth plugin.

For Azure, I can't seem to get a valid token and I get an error on sending email (no wonder if the token is not valid, but I tried anyways). Look at images image1 and image2.

For Google I tried to set it up as well and got this message (image3). Not sure what is wrong here.

@gabrieljenik. Will look into it. We shifted to email relays in the interim https://forums.limesurvey.org/index.php/forum/development/134875-using-google-workspace-email-relay-for-limesurvey - might work out better in the long-term.

@Indispirit did you manage to look into it?

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=35600