View Issue Details

IDProjectCategoryView StatusLast Update
10017Bug reports[All Projects] Data Entry (non public)public2019-09-23 12:18
Reportersanderarons Assigned Toollehar  
PriorityurgentSeveritymajor 
Status confirmedResolutionfixed 
Product Version2.06+ 
Target VersionFixed in Version 
Summary10017: Data from array is not saved correctly into the database (multiple rand() calls in EM/relevance equation)
Description

Limesurvey fails to save data from array questions.

Steps To Reproduce

1 - welcome page with randomization question (1-3)
2 - add 3 groups
3 - respondents are randomized to one group
4 - in each group there are 3 questions, 1 randomization, and 2 array questions
5 - respondents are randomized into one of the two array questions

Additional Information

Please let me know what else you need to know and I will provide the info ASAP.

My plan was to go live this week. However, after checking the data integrity and running some SPSS syntax I noticed the missing data.

TagsNo tags attached.
Complete LimeSurvey version number (& build)Version 2.06+ Build 150629
I will donate to the project if issue is resolvedYes
BrowserChrome
Database & DB-Version5.5.32 - MySQL Community Server (GPL)
Server OS (if known)UNIX
Webserver software & version (if known)libmysql - mysqlnd 5.0.8-dev - 20102224
PHP Version4.1.2

Activities

sanderarons

sanderarons

2015-11-04 09:54

reporter  

report_error_arrays.lss (48,392 bytes)
sanderarons

sanderarons

2015-11-12 11:46

reporter   ~33554

Reminder sent to: c_schmitz, DenisChenu

Hi,

Is there any way to speed up this bug fix? I am waiting to go live with the survey and SSI (provides my respondents) is also waiting on me. However, if I go live with the current bug, about 2/3 of my data might be lost.

Please let me know what my options are or what the expected timeline is.

Kind regards,
Alexander

DenisChenu

DenisChenu

2015-11-12 12:44

developer   ~33555

Multiple question with same code , maybe something here

I can not reproduce after importing and data seems OK.

sanderarons

sanderarons

2015-11-12 13:33

reporter  

data saving error.jpg (242,255 bytes)
sanderarons

sanderarons

2015-11-12 13:38

reporter   ~33556

I've attached a screenshot of the data as it appears in the database. Of the 19 times I have tested the survey, 11 times the data were not saved completely.

Did you test it multiple times?

I'll change the question names and try again with a new survey. I'll keep you posted.

Thanks for the fast response today.

sanderarons

sanderarons

2015-11-12 13:46

reporter  

report_error_arrays v2.lss (46,327 bytes)
sanderarons

sanderarons

2015-11-12 13:46

reporter  

data saving errorv2.jpg (180,449 bytes)
sanderarons

sanderarons

2015-11-12 13:51

reporter   ~33557

Last edited: 2015-11-12 14:26

View 2 revisions

Ok, so in order to rule out multiple questions with the same codes I have renamed them Q1-Q6. After testing this survey (see attached v2) the saving error is still there (see screenshot v2), this time with only 5 correctly saved responses of the 22 test runs I performed.

I am at a loss here...

Could you check if you can reproduce this error? Test it multiple times?

Thanks

DenisChenu

DenisChenu

2015-11-12 14:40

developer   ~33559

I test 5 times with newtest=Y.

Don't want to track down this one.

sanderarons

sanderarons

2015-11-12 15:42

reporter   ~33560

So what other options are there?

jelo

jelo

2015-11-18 10:29

partner   ~33579

What PHP version is used? 4.1.2 is stated in the ticket.

sanderarons

sanderarons

2015-11-18 10:39

reporter   ~33580

Yes, the phpMyAdmin version is 4.1.2. I'm not very much into IT so I hope this is the same as the PHP version you guys are referring to. If not, let me know where I can find the correct PHP version.

sanderarons

sanderarons

2015-11-29 16:53

reporter   ~33634

Reminder sent to: c_schmitz

Are there any other options that I can try? I am still hoping that someone can come up with a fix or explanation otherwise I will have to resort to other software and re-program the entire survey.

c_schmitz

c_schmitz

2015-12-04 10:00

administrator   ~33668

How did you start the survey each time? What link did you use?

sanderarons

sanderarons

2015-12-04 10:10

reporter   ~33669

I have used the Survey URL as can be found in the survey menu:
http://www.aronsconsultancy.nl/surveys/index.php/136935?lang=nl

In the database it says the ref URL is this:
http://www.aronsconsultancy.nl/limesurvey/index.php/admin/survey/sa/view/surveyid/136935

I have also tested it with:
http://www.aronsconsultancy.nl/surveys/index.php/136935?lang=nl&PID=XXX&psid=YYY

Same results, I still get missing data.

DenisChenu

DenisChenu

2015-12-04 10:16

developer   ~33671

Never newtest=Y or newtest/Y ....

sanderarons

sanderarons

2015-12-04 10:24

reporter   ~33672

I just did 20 surveys with http://www.aronsconsultancy.nl/surveys/index.php/136935?newtest=Y&PID=XXX

still 11 cases with missing data.

sanderarons

sanderarons

2015-12-08 09:46

reporter   ~33727

So how can we fix this?

c_schmitz

c_schmitz

2016-02-01 10:59

administrator   ~34434

Last edited: 2016-02-01 11:21

View 3 revisions

I can reproduce this using report_error_arrays v2.lss
This seems only to happen if you check the responses at the same time in the administration repeatedly.

ollehar

ollehar

2016-02-01 12:35

administrator   ~34438

I can reproduce this issue when I do nested conditions like in report_error_arrays v2.

ollehar

ollehar

2016-02-01 14:30

administrator   ~34441

I think the error is that the relevance equation is evaluated more than once, so it's random if the two equations equal each other. Will try to confirm.

You can set the seed of the rand function using srand, e.g. srand(1), in survey index class. For me, it then fails every time to save the response.

ollehar

ollehar

2016-02-01 14:31

administrator  

limesurvey_survey_945277.lss (16,480 bytes)
ollehar

ollehar

2016-02-01 16:30

administrator   ~34444

Basically confirmed that this is the problem. If you disable re-validation in the JumpTo function, this error does not appear.

Solution 1: Pre-process all expressions in the survey and replace rand with the number. This way, rand won't be called twice for one expression.

BUT: Won't work, because where would we save the processed expressions so the survey won't read the original after reload? $_SESSION?

DenisChenu

DenisChenu

2016-02-01 17:07

developer   ~34447

Last edited: 2016-02-01 17:10

View 2 revisions

When user are allowed to "com back" for setting alea, i use:
An short text question, hidden: ALEA
A equation:
{ALEA=if(is_empty(ALEA.NAOK),rand(1,10),ALEA.NAOK)}

This usage is really a specific usage , and sometimes : have {rand(1,10)} in equation question type are needed for some user (please : don't preprocess : this break some other system .....)

See another example on manual: https://manual.limesurvey.org/Expression_Manager_Sample_Surveys#Randomly_Ask_One_Question_Per_Group

With , in Equation : if(EQUA > 1 , EQUA,rand())

ollehar

ollehar

2016-02-01 18:03

administrator   ~34450

Last edited: 2016-02-01 18:04

View 2 revisions

OK, I have a suggestion for a solution where the result of the rand expression is saved in a cache:

                                // Get unique hash for this rand usage, from question id, group id, params and tokens
                                 $flattened_tokens = implode(flattenrec($this->RDP_tokens));
                                 $randHash = hash('md5', $this->questionSeq . $this->groupSeq . $params[0] . $params[1] . $flattened_tokens);

                                 // Create cache if it's not there
                                 if (!isset($_SESSION['randResultCache']))
                                 {
                                     $_SESSION['randResultCache'] = array();
                                 }

                                 // Return cache result if it's there; otherwise, store and return
                                 if (isset($_SESSION['randResultCache'][$randHash]))
                                 {
                                     $result = $_SESSION['randResultCache'][$randHash];
                                 }
                                 else
                                 {
                                     $result = $funcName($params[0], $params[1]);
                                     $_SESSION['randResultCache'][$randHash] = $result;
                                 }
                                 // Cache is cleared in function killSurveySession

This is in em_core_helper.php, lines 2080 ff.

What do you think?

DenisChenu

DenisChenu

2016-02-01 18:37

developer   ~34451

You mean random order ?

This one is tricky : because:

  • In admin : never have random order (else : browse or export are different each time)
  • In public : must be in $_SESSION , or save somewhere in DB

It's the function createFieldMap https://github.com/LimeSurvey/LimeSurvey/blob/master/application/helpers/common_helper.php#L1784 , right ?

I always think : first part must be in cache (and not in session) : https://bugs.limesurvey.org/view.php?id=7988

Randomization etc ... must be done in a plugin : https://github.com/LimeSurvey/LimeSurvey/blob/master/application/helpers/common_helper.php#L2338

Remind createFieldMap is used in admin and in public ....

We have a lot of issue with randomization , with less system in survey ....

Did the Expression system : [if(ALEA > 1,ALEA,rand(1,10)) work ?

ollehar

ollehar

2016-02-02 10:58

administrator   ~34460

Fix committed to bug10017 branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=17040

ollehar

ollehar

2016-02-02 11:02

administrator   ~34461

I pushed my solution to a new branch, bug10017. To be tested and merged later, I guess.

DenisChenu

DenisChenu

2016-02-02 11:23

developer   ~34462

I already say :

  • There are alternate existing solution

But here : you add another function, just need manual update ;)

Maybe unset $_SESSION['randResultCache']) must be in $SESSION['survey'.$iSurveyID] because some user can start (and it work) 2 different survey at a time.

Denis

Related Changesets

LimeSurvey: bug10017 a2dc8abc

2016-02-02 09:57:28

ollehar

Details Diff
Fixed issue 10017: Make sure rand is only evaluated once each survey, using a cache for the function result Affected Issues
10017
mod - application/helpers/expressions/em_core_helper.php Diff File
mod - application/helpers/expressions/em_manager_helper.php Diff File
mod - application/helpers/frontend_helper.php Diff File

Issue History

Date Modified Username Field Change
2015-11-04 09:54 sanderarons New Issue
2015-11-04 09:54 sanderarons File Added: report_error_arrays.lss
2015-11-12 11:46 sanderarons Note Added: 33554
2015-11-12 12:44 DenisChenu Note Added: 33555
2015-11-12 13:33 sanderarons File Added: data saving error.jpg
2015-11-12 13:38 sanderarons Note Added: 33556
2015-11-12 13:46 sanderarons File Added: report_error_arrays v2.lss
2015-11-12 13:46 sanderarons File Added: data saving errorv2.jpg
2015-11-12 13:51 sanderarons Note Added: 33557
2015-11-12 14:26 sanderarons Note Edited: 33557 View Revisions
2015-11-12 14:40 DenisChenu Note Added: 33559
2015-11-12 15:42 sanderarons Note Added: 33560
2015-11-18 10:29 jelo Note Added: 33579
2015-11-18 10:39 sanderarons Note Added: 33580
2015-11-29 16:53 sanderarons Note Added: 33634
2015-12-04 10:00 c_schmitz Note Added: 33668
2015-12-04 10:01 c_schmitz Assigned To => c_schmitz
2015-12-04 10:01 c_schmitz Status new => feedback
2015-12-04 10:10 sanderarons Note Added: 33669
2015-12-04 10:10 sanderarons Status feedback => assigned
2015-12-04 10:16 DenisChenu Note Added: 33671
2015-12-04 10:24 sanderarons Note Added: 33672
2015-12-08 09:46 sanderarons Note Added: 33727
2016-02-01 10:54 c_schmitz Assigned To c_schmitz => ollehar
2016-02-01 10:59 c_schmitz Note Added: 34434
2016-02-01 11:18 c_schmitz Note Edited: 34434 View Revisions
2016-02-01 11:21 c_schmitz Note Edited: 34434 View Revisions
2016-02-01 12:35 ollehar Note Added: 34438
2016-02-01 14:30 ollehar Note Added: 34441
2016-02-01 14:31 ollehar File Added: limesurvey_survey_945277.lss
2016-02-01 16:30 ollehar Note Added: 34444
2016-02-01 16:49 ollehar Status assigned => acknowledged
2016-02-01 17:07 DenisChenu Note Added: 34447
2016-02-01 17:10 DenisChenu Note Edited: 34447 View Revisions
2016-02-01 18:03 ollehar Note Added: 34450
2016-02-01 18:04 ollehar Note Edited: 34450 View Revisions
2016-02-01 18:37 DenisChenu Note Added: 34451
2016-02-02 10:58 ollehar Changeset attached => LimeSurvey bug10017 a2dc8abc
2016-02-02 10:58 ollehar Note Added: 34460
2016-02-02 10:58 ollehar Resolution open => fixed
2016-02-02 11:02 ollehar Note Added: 34461
2016-02-02 11:23 DenisChenu Note Added: 34462
2016-02-04 17:11 ollehar Summary Data from array is not saved correctly into the database => Data from array is not saved correctly into the database (multiple rand() calls in EM/relevance equation)
2019-09-23 12:18 c_schmitz Status acknowledged => confirmed