View Issue Details

This bug affects 3 person(s).
 276
IDProjectCategoryView StatusLast Update
15664Bug reportsSecuritypublic2023-01-25 13:25
ReporterIndispirit Assigned Togabrieljenik  
PriorityimmediateSeverityfeature 
Status assignedResolutionopen 
Summary15664: OAuth support for SMTP & IMAP
Description

Implement OAuth for LS email settings

Email providers are shifting to OAuth to allow app access so requesting that LS support OAuth in email settings.

Example: LS currently accesses Gsuite from Google by taking the login details of a Gsuite account in LS global email settings. Gsuite classifies applications that do this as 'LSAs' - Less Secure Applications and to allow LS to access Gsuite this way currently, you have to provide an extra permission.

Google plans to turn off access for LSAs and only permit access via OAuth: "

Access to LSAs will be turned off in two stages:

1. June 15, 2020 - Users who try to connect to an LSA for the first time will no longer be able to do so. This includes third-party apps that allow password-only access to Google calendars, contacts, and email via protocols such as CalDAV, CardDAV and IMAP. Users who have connected to LSAs prior to this date will be able to continue using them until usage of all LSAs is turned off.
2. February 15, 2021 - Access to LSAs will be turned off for all G Suite accounts."
TagsNo tags attached.
Bug heat276
Complete LimeSurvey version number (& build)
I will donate to the project if issue is resolved
Browser
Database type & version
Server OS (if known)
Webserver software & version (if known)
PHP Version

Users monitoring this issue

User List DenisChenu, Leithy, netikja

Activities

DenisChenu

DenisChenu

2019-12-18 16:01

developer   ~55069

It's one of the reason to rewrite mail system in 4 and extend clearly PHPMailer.
https://github.com/LimeSurvey/LimeSurvey/blob/3ca1be07878d3e3953a7aa38bf7fb5f85345ccb8/application/core/LimeMailer.php#L495
Sample plugin : https://github.com/LimeSurvey/LimeSurvey/tree/develop/application/core/plugins/mailSenderToFrom (more easy this one)

Leithy

Leithy

2022-05-18 14:51

reporter   ~69782

According to this http://www.pmail.com/newsflash.htm Google will require Oauth2 by the end of May 2022 - this month. Would it be worth having this issue bumped up the priority list?

DenisChenu

DenisChenu

2022-05-18 14:56

developer   ~69784

The cost of the assessment typically varies between $10,000 -$75,000 (or more) depending on the size and complexity of the application; smaller applications may see costs at a lower threshold of $4,500.

leave google …

DenisChenu

DenisChenu

2022-05-18 14:56

developer   ~69785

And again : it can be done in plugin

Leithy

Leithy

2022-05-19 13:09

reporter   ~69846

"...leave Google"

Lol...

"...can be done in plugin": Most certainly but many of the existing features could be plugins, including all the SMTP settings. I think a more nuanced rule is used in practice ;)

image.png (9,169 bytes)   
image.png (9,169 bytes)   
DenisChenu

DenisChenu

2022-05-19 14:15

developer   ~69852

Did you really read the link

"Every app that requests access to restricted scope Google user´s data … may see costs at a lower threshold of $4,500."

I confirm : leave google.

Leithy

Leithy

2022-05-19 14:20

reporter   ~69853

Irrelevant to this issue (relates to Android app publishing)?

LimeSurvey doesn't seem to be leaving Google - see image for three Google specific settings in Global Configuration.

NB Oauth2 isn't even just for Google, it's a standard (rammed through by Google) for what that's worth

c_schmitz

c_schmitz

2022-05-20 09:15

administrator   ~69897

So according the schedule Google has deactivated LSA a long time ago.
I did not hear from our user base so far that sending invitations using Gmail would not work.
So, it is not possible to use Google Mail with LimeSurvey now?

Leithy

Leithy

2022-05-20 11:25

reporter   ~69906

I think they planned to do it 2020 but delayed it due to the pandemic.

Seems that it the switchover date for mandatory Oauth2 is now 30 May 2022:

https://support.google.com/mail/thread/153515128/which-email-client-will-work-with-gmail-after-30th-may-2022?hl=en
https://github.com/k9mail/k-9/issues/5956
https://www.ghacks.net/2022/04/06/thunderbird-91-8-0-makes-important-changes-to-google-mail-accounts/

Leithy

Leithy

2022-05-20 11:42

reporter   ~69907

Found the actual announcement on LSAs:

https://support.google.com/accounts/answer/6010255

gabrieljenik

gabrieljenik

2022-10-20 21:49

manager   ~72371

Last edited: 2022-10-20 21:49

Please find below some comments about the analysis of the ticket

About the protocol

https://stackoverflow.com/questions/65228456/how-does-phpmailer-use-oauth/65230616#65230616

What's difficult about XOAUTH2 is obtaining that access token. To get hold of it requires that you run an HTTP service (on top of whatever other services you are running) and provide it with the correct ID and secret, which you submit and receive an access token in response, which you then use in your XOAUTH2 request. PHPMailer provides a script that helps you to do this for Google, Microsoft and Yahoo services, and others via the numerous adapters that are available for the PHPLeague's OAuth2 client package.

IMHO, OAuth2's monumentally confusing structure, ambiguity, and complexity creates excessive overhead and many opportunities for error, and it's all ultimately secured by a username and password, all of which undermines the security of the whole exercise. What doesn't help is that there is no standardisation, and the procedures you must follow to obtain tokens can be completely different across providers (hence the need for different adapters), and is subject to frequent change, making it hard to document. However, since Google imposes it on everyone, you have no choice but to go along with it.

Analysis

Please find below some comments about the analysis done:

Library

For each OAuth2 provider a specific library is needed

Data

In general, I understand that to send the mail it will be needed a client_id, a client_secret, and the Refresh Token.
In the case of Azure you also need a tenantId.
I don't know how it will be with other providers.

Endpoint for getting the Refresh Token

The Refresh Token is assigned by the provider.
An endpoint is required to receive it from the provider and save it.

  • It has to be HTTPS
  • The URL is configured in the Google console (or the provider setting).

UI for Refresh Token

Global Settings > Email Settings should be updated

  • It should have some warning that OAUTH only works if HTTPS is enabled.
  • It should ask you:
    • Provider (Google, Microsoft, Yahoo, etc.)
    • Client ID
    • Client Secret
      When saving (or maybe with a new get refresh token button), it should send you to the provider page to confirm the permissions. The new endpoint would receive the message and save the refresh token in a global setting (no UI).

We could also fetch the settings with a plugin event.

Plugin event

Ideally there would be new Plugin Events added so that plugins can register new providers.
The provider is used in two moments:

When fetching the refresh token

We could implement the UI for refresh token as plugin settings.
This common behaviour could be on a BaseSMTPOauth class.

When sending the mail.

To get the AbstractProvider instance that PHPMailer needs, additional parameters may be needed (such as the tenantId in the case of Azure), so somehow the plugin would have to indicate what additional fields should be add in Global Settings.

Also, we could set the provider on a plugin event (ex: beforeEmail)

DenisChenu

DenisChenu

2022-10-21 08:45

developer   ~72372

Don't try to develop OAuth system,
PHPMailer already have solution : https://github.com/LimeSurvey/LimeSurvey/blob/master/application/third_party/phpmailer/src/OAuth.php

The doc is here …
https://github.com/PHPMailer/PHPMailer/wiki/Gmail-XOAUTH2-Using-Google-API-Client

I really think it must be a Plugin , core plugin yes, but a plugin.

gabrieljenik

gabrieljenik

2022-10-21 13:59

manager   ~72373

Don't try to develop OAuth system,

Never intended to

DenisChenu

DenisChenu

2022-10-21 14:09

developer   ~72374

Never intended to

Seems need specific library for GMail

composer require google/apiclient ^2.11

Then really better in a plugin

c_schmitz

c_schmitz

2022-10-26 10:22

administrator   ~72428

I would suggest that as a first step you add OAUTH support for Google as described in the PHPMailer docs.
After that is implemented we will see how we go from there.

gabrieljenik

gabrieljenik

2022-10-26 15:40

manager   ~72436

I would suggest that as a first step you add OAUTH support for Google as described in the PHPMailer docs.

You mean a PoC?
Or adding that to Lime with all the configuration screens?

If you intended the latter, I guess you are OK with the analysis and approach, right?

DenisChenu

DenisChenu

2022-10-26 17:58

developer   ~72440

Or adding that to Lime with all the configuration screens?

A lot of configuration screen + a complete library

A good reason to set it as Plugin ?
No ?
@c_schmitz ?

gabrieljenik

gabrieljenik

2023-01-03 20:50

manager   ~73340

Picking this up. As a plugin :)

gabrieljenik

gabrieljenik

2023-01-09 17:54

manager   ~73405

Hi All,

Let me show you how about the progress on this story.

We have developed a core plug-in that:

1- Pickups the credentials for connecting to Google services.

2 - Calls Google services to get a refresh-token, that will be used when sending emails.

3 - Intervene during the email sending process to avoid sending the emails through the regular means and send them directly through Google services.

Please find below a few screenshot.

Working on this proof-of-concept we have come with the following issues or observations.

Https required

Google requires an https URL callback when setting up the service. So HTTP will be required for using this plugin. We will be adding I checked on that as well as a warning if not having the proper schema.

From Address

Google seems to overwrite the from address and use the one related to the account.

New Email Events

Right now we only have beforeEmail event, which is used generally for decorating the email (Ex. adding a header), or for sending it through a different non-standard mean.

As the plug-ins don't have a synchronization mechanism, if there are two plug-ins using the same beforeEmail event, is hard to know for sure which plug-in will run first and which will run later.

Of that we believe we need a specific event named beforeSendingEmail. This way we will be able to separate two different processes:

A - Decorating the email (as adding a header or setting specific form addresses depending on the survey)

B - Sending it (as sending it through SMTP, OAuth, or even other gateways).

Global setting screen

We will need to inform the administrators that the OAuth plugin is active. If not administrator could get confused and think that the emails will be sent through regular means.

Right now for this proof-of-concept we are going to be doing it through JavaScript on an event. We believe we may need something a little bit more robust as there could be many other email plug-in.

We don't have a clear view on this but we picture having an email plugin coordinator, which can query the different email plug in and, for example, identify if the emails will be sent to a regular mean or through a plug-in email.

The global settings screen could use this new blocking email coordinator for knowing if the SMTP credential fields should be disabled or not, and also for showing a text message that says which plug-in will be handling emails.

Porting code to new plugins

Looking to the Future the current meaning for sending emails could be relocated in one or many core plug ins.

image-2.png (233,769 bytes)
image-3.png (312,429 bytes)
DenisChenu

DenisChenu

2023-01-09 18:02

developer   ~73406

Last edited: 2023-01-09 18:07

As the plug-ins don't have a synchronization mechanism, if there are two plug-ins using the same beforeEmail event, is hard to know for sure which plug-in will run first and which will run later.
Of that we believe we need a specific event named beforeSendingEmail. This way we will be able to separate two different processes:

beforeEmail receive whole LimeMailer (extend PHPMailer) as ->get('mailer'), then it can be used for anything that is related to mail in my opinon.

You can use priority in config.xml to be sure to use this way to send email after all olther are done. It's done for this.

But : we always have incompatibility issue with other plugin.
For example : you can not use https://github.com/Xites/LS_mailDKIM with google.

Or : developer can not really be sure other plugin broke somthing : https://gitlab.com/SondagesPro/mailing/noMailToExample must happen before google here for example.

gabrieljenik

gabrieljenik

2023-01-24 21:58

manager   ~73551

POC: https://github.com/LimeSurvey/LimeSurvey/tree/feature/15664-Google-OAuth-plugin

DenisChenu

DenisChenu

2023-01-25 07:53

developer   ~73554

Great !

Maybe PR as draft ? Hard to compare here

gabrieljenik

gabrieljenik

2023-01-25 13:25

manager   ~73556

https://github.com/LimeSurvey/LimeSurvey/pull/2868

Issue History

Date Modified Username Field Change
2019-12-17 12:50 Indispirit New Issue
2019-12-18 16:00 DenisChenu Issue Monitored: DenisChenu
2019-12-18 16:01 DenisChenu Note Added: 55069
2022-05-18 14:51 Leithy Note Added: 69782
2022-05-18 14:51 Leithy Bug heat 254 => 256
2022-05-18 14:52 Leithy Issue Monitored: Leithy
2022-05-18 14:52 Leithy Bug heat 256 => 264
2022-05-18 14:56 DenisChenu Note Added: 69784
2022-05-18 14:56 DenisChenu Note Added: 69785
2022-05-19 13:09 Leithy Note Added: 69846
2022-05-19 13:09 Leithy File Added: image.png
2022-05-19 14:15 DenisChenu Note Added: 69852
2022-05-19 14:20 Leithy Note Added: 69853
2022-05-20 09:15 c_schmitz Note Added: 69897
2022-05-20 09:15 c_schmitz Bug heat 264 => 266
2022-05-20 11:25 Leithy Note Added: 69906
2022-05-20 11:42 Leithy Note Added: 69907
2022-09-25 19:25 netikja Issue Monitored: netikja
2022-09-25 19:25 netikja Bug heat 266 => 274
2022-10-18 17:00 c_schmitz Priority none => immediate
2022-10-18 17:00 c_schmitz Assigned To => gabrieljenik
2022-10-18 17:00 c_schmitz Status new => assigned
2022-10-18 17:01 c_schmitz Summary [feature request] OAuth implementation => OAuth support for SMTP & IMAP
2022-10-18 17:01 c_schmitz Project Feature requests => Bug reports
2022-10-20 21:49 gabrieljenik Note Added: 72371
2022-10-20 21:49 gabrieljenik Bug heat 274 => 276
2022-10-20 21:49 gabrieljenik Note Edited: 72371
2022-10-21 08:45 DenisChenu Note Added: 72372
2022-10-21 13:59 gabrieljenik Note Added: 72373
2022-10-21 14:09 DenisChenu Note Added: 72374
2022-10-26 10:22 c_schmitz Note Added: 72428
2022-10-26 15:40 gabrieljenik Note Added: 72436
2022-10-26 17:58 DenisChenu Note Added: 72440
2023-01-03 20:50 gabrieljenik Note Added: 73340
2023-01-09 17:54 gabrieljenik Note Added: 73405
2023-01-09 17:54 gabrieljenik File Added: image-2.png
2023-01-09 17:54 gabrieljenik File Added: image-3.png
2023-01-09 18:02 DenisChenu Note Added: 73406
2023-01-09 18:07 DenisChenu Note Edited: 73406
2023-01-24 21:58 gabrieljenik Note Added: 73551
2023-01-25 07:53 DenisChenu Note Added: 73554
2023-01-25 13:25 gabrieljenik Note Added: 73556