Relationship Graph

Relationship Graph
related to related to child of child of duplicate of duplicate of

View Issue Details

This bug affects 1 person(s).
 18
IDProjectCategoryView StatusLast Update
16330Bug reportsEncryptionpublic2021-07-12 11:53
Reporterpns1953 Assigned Top_teichmann  
PrioritynoneSeveritypartial_block 
Status closedResolutionfixed 
Product Version4.2.4 
Fixed in Version5.x 
Summary16330: Encrypted fields not decrypted correctly when viewing responses using web UI or when exported to .lsa
Description

Survey includes one question where encryption has been specified (a multiple short text question requesting Firstname, Lastname).
There are no errors when respondents submit, and the database field is showing ciphertext for responses to this question.
However when responses are examined using the Web UI, or exported in a .lsa file, they are NOT shown correctly decrypted but in what also appears to be ciphertext (but is NOT the same character sequence as in the database, so some attempt at decryption seems to be happening).

The security.php file has not been changed during the currency of this survey.

Steps To Reproduce

1) Import and activate attached survey, which has 16 questions in 4 groups.
2) Complete one response - apart from the first question, null responses can be given to most questions but enter text for [G01Q07] "Please give your name"
3) View response for this entry - field is not decrypted as expected.

Additional Information

1) A simple test survey with one encrypted question of the same type (no other questions in survey) does NOT reproduce the problem - the fields in the question are decrypted correctly.

2) I don't think this is relevant, but "for info" the survey exhibiting the bug, and also the simple test survey noted here, have been moved between LimeSurvey instances - behaviour is the same for each survey regardless of server (other server is on 4.2.3)

TagsNo tags attached.
Attached Files
Bug heat18
Complete LimeSurvey version number (& build)4.4.11 git
I will donate to the project if issue is resolvedNo
BrowserSafari, Chrome
Database type & versionMariaDB 10.3.23
Server OS (if known)linux
Webserver software & version (if known)LiteSpeed
PHP Version7.2

Relationships

related to 15669 feedbackp_teichmann Crypted data are not exported uncrypted 
related to 15668 closedp_teichmann Crypted data are not shown uncrypted 
related to 15670 closedp_teichmann Reloading response via token : crypted data 
related to 16971 closedp_teichmann Unable to use of TOKEN attribute encrypted during survey (expression manager) 

Activities

DenisChenu

DenisChenu

2020-05-26 10:06

developer   ~58020

About LSA export : i think it's great if we have a "LSA decrypted" export, then user can choose if done for backup for same server then can be crypted. Or not.

:)

The worst one are Reloading response via token : crypted data

DenisChenu

DenisChenu

2020-05-26 10:09

developer   ~58021

Last edited: 2021-01-12 17:35

1) A simple test survey with one encrypted question of the same type (no other questions in survey) does NOT reproduce the problem - the fields in the question are decrypted correctly.

? Are you sure ?

Try to import included lsa : i think it broke with a 500 error (Wrong decryption key! Decryption key has changed since this data were last saved, so data can't be decrypted. Please consult our manual at https://manual.limesurvey.org/Data_encryption#Errors.)

pns1953

pns1953

2020-05-26 11:42

reporter   ~58022

Last edited: 2021-01-12 17:35

Hi - I think I may have diverted attention away from my main issue here by talking about exporting / importing survey archives with encrypted fields.
I did spot issues 15668 and 15669 before posting, and have verified in a one-question survey with encryption that in this case, my export is decrypted - obviously that will be necessary to migrate it to another server with different encryption keys. That may not be so wise for a server backup but that's another question for debate / feature request to select as part of the export.

I did import your test survey and that had an error as expected when I tried to display a response:
"500: Internal Server Error
Call to a member function decrypt() on null"

This is my issue:
With the survey I uploaded here, I get NO errors when I activate the survey, enter one response including 'First Name, Surname', and then display the response.
In the DB I can see that the relevant fields are encrypted, but when displaying the response I see:

[G01Q07_SQ01] (First name) Ys1jwmKfS1+rYB274j2gO06GPrYdL...
[G01Q07_SQ02] (Surname) 5PORCDiCzCidyf4hN0LbX0wdG74d...

(I've shortened the strings above). No error is shown but the decryption is incorrect (the above strings are different from the DB fields for those questions)
Are you able to replicate the problem with my survey?
There seems to be something specific in my uploaded survey that's stopping the correct decryption, though the simple one-question survey with encryption DOES work.
Thanks!

DenisChenu

DenisChenu

2020-05-26 11:49

developer   ~58023

Last edited: 2021-01-12 17:35

I can confirm i have encrypted field in browse response.

The strange part : « though the simple one-question survey with encryption DOES work.» : you mean a survey with only ONE question (encrypoted) ?

Because a survey with 2 question : one crypted + on not crypted show (and export) only crypted data.

DenisChenu

DenisChenu

2020-05-26 11:49

developer   ~58024

Last edited: 2021-01-12 17:35

DenisChenu

DenisChenu

2020-05-26 11:52

developer   ~58025

Last edited: 2021-01-12 17:35

OK : database are not the same, but export as LSA, import as LSA : work and show same data.

pns1953

pns1953

2020-05-26 12:02

reporter   ~58026

Last edited: 2021-01-12 17:35

Yes that seems to replicate what I'm seeing - both web UI and exported LSA are showing the same data (WRONGLY decrypted version of the DB contents).

Re your previous note - I have made two test surveys - one with only one question that was encrypted, the second survey with two questions, one of which was encrypted. BOTH behaved as expected with the DB fields being encrypted, but shown correctly in the Web UI and being exported unencrypted (as in the attached LSA for the two-question survey).
It is strange that the problem seems to be present in some surveys but not others. If if you are seeing the problem in your survey with only two questions (one encrypted) that may make debugging easier hopefully.

DenisChenu

DenisChenu

2020-05-26 12:31

developer   ~58028

Last edited: 2021-01-12 17:35

It's not really wrongly encryted since we can export as lsa and import (in same instance) without issue.

If there are a bad system here : we can not do it :)

pns1953

pns1953

2020-05-26 12:59

reporter   ~58032

Last edited: 2021-01-12 17:35

Yes I understand what you are saying, the decryption on the 'bad' surveys / fields isn't showing an error so must the actual decryption must be happening correctly, but if so, then the correct plaintext string resulting from that decryption isn't being shown in the UI or the LSA. I have checked to see if a small sub-string of what is displayed in the UI / LSA field is present in the encrypted DB field, and it isn't. So I'm not sure where the output shown in UI / LSA is coming from.

GuernseyResearch

GuernseyResearch

2020-06-30 18:35

reporter   ~58599

Last edited: 2021-01-12 17:35

Further information:
When a person drops out of the survey without a submit the encrypted data appears fine in the show responses for that record.
However, when the person completes the survey with a submit at the end the encrypted data is not properly displayed in the UI (show responses) for that record.

GuernseyResearch

GuernseyResearch

2020-07-09 00:58

reporter   ~58817

Last edited: 2021-01-12 17:35

To follow with more examples - I have a short survey done on Version 4.3.3+200707. An export is attached.
The two questions "appear" to be the same and both are encrypted. The email that is sent to the admin shows with answers without clear text for either (attached)
The Admin "show responses" shows the first question garbled, but the second question is properly decrypted in the UI (response1)

You can create multiple questions of different types and some will be correct and others are not.
IMPORTANT - changing the ORDER of the questions has an impact. (response2) as the first is wrong and the second correct.
Adding another question appears to work properly. (response3)

email.txt (1,201 bytes)   
Email that gets sent to admin  - encryted fields are "hidden"


Hello,

A new response was submitted for your survey 'testing'.

Click the following link to see the individual response:
https://guernseyresearch.com/survey/index.php/admin/responses/sa/view/surveyid/274463/id/11

Click the following link to edit the individual response:
https://guernseyresearch.com/survey/index.php/admin/dataentry/sa/editdata/subaction/edit/surveyid/274463/id/11

View statistics by clicking here:
https://guernseyresearch.com/survey/index.php/admin/statistics/sa/index/surveyid/274463


The following answers were given by the participant:


     Response ID : 11
     Date submitted : 01-01-1980 00:00:00
     Last page : 3
     Start language : en
     Seed : 1238572615

My first question group

     <p>What color</p> : Red [AO01]
     <p>email</p> : aKcreBSO7Gov58aIL6MkyrOn8ECq3BWpSsxBYIyR4i6UBq2DjkXV50+Z7PLLHwihHxXLwCATrHvuE7LzycJiBWQvblMxbUJzZEtVaDE3WktaV2M3TGZ6V1dLRThxM3ZDRWhMZnJQUVBpRTJ0bEp5WmkzNllCeEh5NGxsWkNUaThnTmNNaGVqWmsremFVR0RYZzNGRkRXUmhkbVZBWldVPQ==
     <p>name</p> : 1su1M7yg9GwdTt2P4mSCWTCYnJIwECbYcWUkX3CLd0y4mNPSI5IF8DhdxODp1KdrXiDilOlWAbPSO7OzBfmtBURhdmU=
email.txt (1,201 bytes)   
Response1.jpg (142,187 bytes)
Response2.jpg (91,899 bytes)   
Response2.jpg (91,899 bytes)   
Response3.jpg (94,834 bytes)   
Response3.jpg (94,834 bytes)   
DenisChenu

DenisChenu

2020-09-16 19:29

developer   ~59855

Last edited: 2021-01-12 17:35

Checked both : seems fixed now (export lsa from one system to another instance)

DenisChenu

DenisChenu

2020-10-26 14:51

developer   ~60412

Last edited: 2021-01-12 17:35

@GuernseyResearch : i confirm the 2 issue on last master.

Can you report separatly maybe ? @cdorin : need 2 issue or can be fixed here ?

GuernseyResearch

GuernseyResearch

2021-01-12 17:30

reporter   ~61502

Last edited: 2021-01-12 17:35

This appears to be fixed in Version 4.4.0-RC3+210112

ollehar

ollehar

2021-01-12 17:35

administrator   ~61503

Last edited: 2021-01-12 17:35

Fixed

DenisChenu

DenisChenu

2021-03-05 08:05

developer   ~62744

However when responses are examined using the Web UI

Open again

DenisChenu

DenisChenu

2021-03-05 08:07

developer   ~62745

No need token : just

  1. Import
  2. Activate
  3. Launch
  4. Submit
  5. Browse, view and edit
xieve_

xieve_

2021-03-28 16:31

reporter   ~63663

Having the same problem with a few forms that were conducted before 4.4.0 was out (some time last month, I believe). The encryption was only enabled on some fields, probably by accident, and only these fields are being shown non-decrypted. I tried copying the database and setting up a new LimeSurvey instance pulled directly from master with the old security.php - no luck. Issue persists across Web UI and all exports, which makes it impossible for us to extract any of the answers. As some of those surveys were pretty critical and the results are going to be needed soon, is there any way that they could possibly be decrypted "manually"? Do I understand correctly that AES-256-GCM is used for encryption? Is there any other information I can provide or something I should try?

p_teichmann

p_teichmann

2021-05-21 10:29

administrator   ~64504

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&amp;id=31818

Mazi

Mazi

2021-06-25 10:31

updater   ~65058

@GuernseyResearch, can you confirm that this problem is fixed for you at the latest version?

Did you notice any performance issues when exporting encrypted data of a larger survey (our survey has ~300 questions and 1000 responses)?

GuernseyResearch

GuernseyResearch

2021-06-25 16:31

reporter   ~65064

Our key survey that uses encryption has 40 questions (some are skipped) and TWO fields near the end that are encrypted because we ask for an optional name and email address if they want to participate in another survey. We have close to 1000 responses and don't appear to have a problem. When we do an export of the results we don't need to see the encrypted data (that is done as we process each response and communicate individually with the people responding). As a result, we have not seen any performance issues in the actual survey data.

Mazi

Mazi

2021-06-25 22:02

updater   ~65065

Thanks for sharing your feedback!

Mazi

Mazi

2021-06-25 22:05

updater   ~65066

We have many encrypted fields (~300 questions) and 1600 responses and the screen just freezes on export.

When exporting a smaller amount of responses, we get empty or "n/a" at the responses, though there IS some data. But that may be related to testing at a different system and not using the correct key for decryption... not sure. Maybe anyone has seen those emtpy responses at the export as well?

GuernseyResearch

GuernseyResearch

2021-06-25 22:18

reporter   ~65067

Do you need ALL those fields to be encrypted?
We are trying to follow privacy rules and protect personal information (name, email). As long as those are blocked the other data is not associated with a person. The database itself is protected so we feel that we are adequately following the requirements.
What causes you to encrypt so much of the survey?

DenisChenu

DenisChenu

2021-06-26 16:49

developer   ~65104

not using the correct key for decryption...

Sure it broke somewhere here …

c_schmitz

c_schmitz

2021-07-12 11:53

administrator   ~65280

Release done.

Related Changesets

LimeSurvey: master bd1ae3af

2021-05-18 10:11:13

p_teichmann

Details Diff
Fixed issue 15670, 16330: loaded survey responses are not being decrypted Affected Issues
15670, 16330
mod - application/helpers/frontend_helper.php Diff File

Issue History

Date Modified Username Field Change
2020-05-25 14:55 pns1953 New Issue
2020-05-25 14:55 pns1953 File Added: limesurvey_survey_597794.lss
2020-05-26 10:04 DenisChenu Relationship added related to 15669
2020-05-26 10:04 DenisChenu Relationship added related to 15668
2020-05-26 10:04 DenisChenu Relationship added related to 15670
2020-05-26 10:06 DenisChenu Note Added: 58020
2020-05-26 10:09 DenisChenu Note Added: 58021
2020-05-26 10:09 DenisChenu File Added: survey_archive_crypted.lsa
2020-05-26 11:42 pns1953 Note Added: 58022
2020-05-26 11:49 DenisChenu Note Added: 58023
2020-05-26 11:49 DenisChenu Note Added: 58024
2020-05-26 11:49 DenisChenu File Added: Capture d’écran du 2020-05-26 11-47-31.png
2020-05-26 11:52 DenisChenu Note Added: 58025
2020-05-26 11:52 DenisChenu File Added: Capture d’écran du 2020-05-26 11-51-21.png
2020-05-26 11:52 DenisChenu File Added: Capture d’écran du 2020-05-26 11-50-34.png
2020-05-26 12:02 pns1953 Note Added: 58026
2020-05-26 12:02 pns1953 File Added: survey_archive_971865-2.lsa
2020-05-26 12:31 DenisChenu Note Added: 58028
2020-05-26 12:59 pns1953 Note Added: 58032
2020-05-27 15:36 ollehar Category Response browsing => Encryption
2020-06-30 18:35 GuernseyResearch Note Added: 58599
2020-07-09 00:58 GuernseyResearch Note Added: 58817
2020-07-09 00:58 GuernseyResearch File Added: limesurvey_survey_274463.lss
2020-07-09 00:58 GuernseyResearch File Added: email.txt
2020-07-09 00:58 GuernseyResearch File Added: Response1.jpg
2020-07-09 00:58 GuernseyResearch File Added: Response2.jpg
2020-07-09 00:58 GuernseyResearch File Added: Response3.jpg
2020-08-07 19:54 kjnerhus Issue Monitored: kjnerhus
2020-09-16 19:29 DenisChenu Note Added: 59855
2020-10-26 14:51 DenisChenu Note Added: 60412
2020-10-28 18:28 LimeBot Sync to Zoho Project => |Yes|
2020-10-28 18:45 LimeBot Sync to Zoho Project Yes => |Yes|
2020-10-28 18:49 LimeBot Sync to Zoho Project Yes => |Yes|
2021-01-12 17:30 GuernseyResearch Note Added: 61502
2021-01-12 17:35 ollehar Assigned To => ollehar
2021-01-12 17:35 ollehar Status new => resolved
2021-01-12 17:35 ollehar Resolution open => fixed
2021-01-12 17:35 ollehar Fixed in Version => 4.4.0-RC2
2021-01-12 17:35 ollehar Note Added: 61503
2021-01-13 08:49 DenisChenu Relationship added related to 16971
2021-01-29 08:23 cdorin Status resolved => closed
2021-03-05 08:05 DenisChenu Status closed => feedback
2021-03-05 08:05 DenisChenu Resolution fixed => reopened
2021-03-05 08:05 DenisChenu Complete LimeSurvey version number (& build) 4.2.4+200520 => 4.4.11 git
2021-03-05 08:05 DenisChenu Note Added: 62744
2021-03-05 08:07 DenisChenu Note Added: 62745
2021-03-05 08:07 DenisChenu File Added: limesurvey_survey_CryptedToken.lss
2021-03-05 08:07 DenisChenu File Added: Capture d’écran du 2021-03-05 08-06-07.png
2021-03-05 08:07 DenisChenu File Added: Capture d’écran du 2021-03-05 08-06-14.png
2021-03-28 16:31 xieve_ Note Added: 63663
2021-03-29 11:09 ollehar Assigned To ollehar => p_teichmann
2021-05-17 11:03 p_teichmann Status feedback => assigned
2021-05-21 10:29 p_teichmann Changeset attached => LimeSurvey master bd1ae3af
2021-05-21 10:29 p_teichmann Note Added: 64504
2021-05-21 10:29 p_teichmann Resolution reopened => fixed
2021-06-25 10:31 Mazi Note Added: 65058
2021-06-25 16:31 GuernseyResearch Note Added: 65064
2021-06-25 22:02 Mazi Note Added: 65065
2021-06-25 22:05 Mazi Note Added: 65066
2021-06-25 22:18 GuernseyResearch Note Added: 65067
2021-06-26 16:49 DenisChenu Note Added: 65104
2021-07-06 09:20 p_teichmann Status assigned => closed
2021-07-06 09:20 p_teichmann Fixed in Version 4.4.0-RC2 => 5.x
2021-07-06 09:26 p_teichmann Status closed => resolved
2021-07-12 11:53 c_schmitz Note Added: 65280
2021-07-12 11:53 c_schmitz Status resolved => closed