Anonymous Login
2017-01-24 06:01 CET

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
05269User patchesConditionspublic2012-06-21 13:22
Assigned ToTMSWhite 
Product Version1.90+ 
Target VersionFixed in Version2.00 
Summary05269: Use ExpressionManager for Branching logic as optional alternative to Conditions
DescriptionThe current Conditions system, although boasting a nice user interface, does not support complex logic, such as parentheses, branching based upon mathematical results or other functions.

Given the installed user base, Conditions should not be replaced.

However, I proposed we give users the option to use ExpressionManager as an alternative to Conditions. The steps are:
(1) Add two additional fields to $ia for each question:
 (a) Relevance - this boolean equation will determine whether the question should be displayed
 (b) VarList - this is the comma-delimited set of variables used within any part of the question (Relevance, substitutions within the question or answer text). The ExpressionManager already computes this list when parsing the equation.
(2) Operationally, for each displayed survey page (whether it is question at at time, group at a time, data entry, of full-survey), LimeSurvey:
 (a) Collects next set of possibly relevant questions (e.g. next group)
 (b) Collects values needed to compute relevance (from VarList)
 (c) Uses ExpressionManager to evaluate the relevance of each question in the set
 (d) If any of the questions are relevant, then display that set of questions
 (e) If none of the questions are relevant, then get the next set of potentially relevant questions and repeat from (2)(b)
 (f) If there are no potentially relevant questions left, then the survey is done.

The ExpressionManager already exists (see Issue 05103 and 05268 for the patch).

So, the steps would be:
(1) Add the two columns to $ia (and extend the database structure to support those columns)
(2) Figure out best way to implement the Operational workflow
  (a) What is best way in LS1?
  (b) What about in CI?
(3) Make this functionality optional? (like Tokens, require user click something to turn it on?)
TagsNo tags attached.
Complete LimeSurvey version number (& build)10256
Attached Files

related to 05103closedTMSWhite Support conditional piping/tailoring and complex calculations via embedded equation parser 
related to 05268closedc_schmitz Do all LimeReplacementField and Token replacements in a single function 
related to 05288closedTMSWhite Optionally replace Assessments with ExpressionManager features 



lemeur (developer)


Given the excellent impression your work gives me, I would like to have your implementation be optionnal at first, but quickly modify the GUI so that your implementation becomes the standard one (removing the old and dirty condition eval code). We'll then let user decides between using a simple GUI which translates graphical input to your expression syntax, or directly an advanced text-oriented expression editor which would let the survey designer use all the power of your Expression engine. This answers point (3) ;-)

Ok for step (1) as well.

(2), Indeed I think we need to synch with CI coders to understand how independant you are from the CI version and how to let you implement on LS1 and yet make it easy for GSoC participant to integrate your mods in the CI version.

==> Sorry, I know I'm of little help, but I try to do my best ;-)



TMSWhite (reporter)

Some notes to myself about this task:

(1) All conditions logic for a page are translated into a single Javascript function called checkconditions(), created in group.php and survey.php

(2) General approach should be to:
(a) First check that each variable is not null and its value is not blank
(b) For any numeric comparator, first do parseFloat()
(c) Then implement each of the boolean equations
(d) If true, use this syntax:
(e) If false, use this syntax to make invisible:

(3) For implementing functions, as long as have PHP and JavaScript versions of each of them, it will work.
(a) Will need to decide whether to use JavaScript, PHP, or other naming convention, as there are definite differences
(b) ExpressionManager already extended to store the mapping of PHP to JavaScript function names. Unknown JavaScript mappings are called NA, so can call a single function to avoid runtime exceptions
(c) Once clear on how to map all SGQA values to JavaScript declarations, can get ExpressionManager to generate the needed JavaScript rules in checkconditions, complete with appropriate functions, nested boolean conditions, math operators, etc.
(d) Main risk is different handling of String=>Numeric conversions by PHP vs. JavaScript. Currently, ExpressionManager doesn't try to figure out whether the operands to an Expression are (both) numeric, string, or date. Will need to test JavaScript to see whether stronger typing is needed.


lemeur (developer)

Some comments:

(3c) qanda.php:function retrieveJSidname($cd,$currentgid=null) will give the mapping between SGQA to DOM elements depending on the current page displayed

For token attributes there is another difficult thing that was implemented: we try to hide the value of the token attribute from the Javascript condition function when possible (for instance if the condition uses a comparizon between a token attribute and a question asked in a previous page, the eval is done on the server side, and only the result 1==1 or 1==2 is pushed to the Javascript checkconditions() function. This prevents disclosing sensitive personnal information to the client-side [for instance medical analysis results]).

(3d) True, this is a common issue. Some questions are assumed to be numerical only and thus we use numerical comparizons. Others may have both kinds of answers, and thus we have sometimes used a question attribute to let the survey designer identify the type of answer given.


TMSWhite (reporter)

Thanks, Thibault.

Out of curiosity, what is the history behind using javaSGQA vs. answerSQGA naming?

If we move to allowing changes of other variables via equations (whether they be on the same, prior, or subsequent pages), will that naming system affect whether or not the responses are actually saved to the database?


lemeur (developer)

I must confess that I don't know the history behind using javaSGQA or answerSGQA because when I first started to work on the conditions code this was hardcoded in survey.php and group.php. I only factorized the code in the retrieveJSidname function.

I first guessed that javaSGQA was used to keep track of answers from previous pages (in case they are used in conditions), and answerSGQA used for answers currently displayed in the page. However, when I worked on the retrieveJSidname function I realized that sometimes javaSGQA is also used for questions displayed in the current page!


lemeur (developer)

There is one thing we need to focus on because current implementation of conditions doesn't handle it well. It is the case of "Chained conditions", that is to say when a condition on question Q3 uses an answer to a previous question Q2 which is itself conditionnal. Currently we require that the survey-designer copy the conditions set to Q2 to Q3, but this is a little difficult to understand for our users.

It would be great to keep track of displayed questions or not so that evaluating chained-conditions is automatically done by LS. But this is not always easy to decide what to do:
* if Q3 must be displayed if "Q2 == Yes", then it is easy to assume that if Q3 is not displayed then Q3 mustn't be displayed
* if Q3 must be displayed if "Q2 != Yes", then it is NOT easy to assume what we expected LS to do if Q3 is not displayed!

Do you have experience on such issues ?


TMSWhite (reporter)

Yes, we dealt with chained conditions all the time. We would often use Equation-type questions to store intermediate values (or complex conditions themselves) to minimize the length of the chained condition.

The main challenge will be "bubbling" of doing the evaluations for the survey style where all questions are shown on one page. So, even if you run checkconditions() with each focus change, some steps within the checkconditions() could affect the values of other steps - so either order becomes important, or we need to find an intelligent way to ensure that all of the conditions are processed before making final decisions about what to hide and display. This may be merely a re-factoring issue for checkconditions(), but we'll want to build some strong unit tests to be sure.

Can you provide some more complex examples of chained conditions you'd hope to support?


TMSWhite (reporter)

In retrospect, the "bubbling" effect should not be a problem as long as people don't put side-effects into their equations (such as using ++ or -- to change variable values).

The following scenario shows a wrong and right way to deal with chained conditions.

About 10 years ago, we implemented the full SCID-II (the American Psychiatric Association's Structured Clinical Interview for Diagnosis for Axis II - Personality Disorders). In it, we first asked the subject to answer 120 questions about symptoms. For each symptom to which the subject attested, a reviewer would ask focused follow-up questions to determine whether the person truly met the criteria for that symptom (another 400 questions). Then, we computed the assessment scale to determine whether the person had enough symptoms to meet the diagnostic criteria.

That worked fine until some epidemiologist colleagues felt that 520 questions was too many. They wanted to ask the minimum number of questions needed to rule in or rule out each diagnosis. So, after the 120 initial questions, we only asked enough follow-up questions per diagnostic category to ensure we had the right diagnosis, then we'd move on to the next.

I was lazy and used ++ and -- to change a counter of the number of validated symptoms. But, as people moved backwards and forwards through the interview, those calculation rows would give the wrong answer (since you'd increment or decrement values each time the question was visited).

Then, we switched it to the right strategy, which was to put the full diagnostic criteria as the relevance for each criteria. It was a little more typing (but using Excel, it was easy to cut and paste), and since it was not using variable assignments (and computed the full assessment score at each step), it was guaranteed to be correct all the time.

So, those are some examples of highly chained conditions, but that it is also possible to reliably implement them as long as you avoid letting one equation assign the value of another variable.


TMSWhite (reporter)

Works for static use (e.g. no on-same-page changes) in limesurvey_dev_tms revision 10515

There is now a "Relevance" Question Attribute for all question types. The default value is 1. If the value is 1 or blank (to make backwards compatabile with existing surveys that don't have this attribute, then Relevance is true, and the question is displayed (unless there is a Condition that would prevent it from being displayed). If the Relevance equation is false, then the question will not be displayed.

Status Update:
(1) Works for already-set values (e.g. on prior or future pages). Does not yet dynamically change question visibility on same page

(2) Currently does not change irrelevant values (e.g. does not mark them as Not Applicable or NULL them in 'fieldarray'.
(a) What is the desired behavior? Should it only NULL values if $deletenonvalues==true?

(3) Currently does not write the question to .html at all if the question is irrelevant (rather than keeping it on page but making it hidden).
(a) Probably need to switch to include value on page, or at least set flag so that save.php can NULL the value if needed.

(4) What happens if there are no relevant questions in a group? Will LimeSurvey advance to the next group?


lemeur (developer)

2a) Nulling irrelevant values while moving forward/backward was the old behaviour with deletenonvalue=true.
However, for performances issues, it was recently changed so that we now only Null irrelevant values when Submitting the final response.

3a) Yes maybe it would be better to at least have the hidden input field displayed because some customer may want to use custom javascript

4) yes, see the checkgroupfordisplay() function. Equally the checkquestionfordisplay is/was used for the deletenonvalue feature.


TMSWhite (reporter)

Thanks, lemeur. You responded at the same time that I posted a similar set of questions here:

The lingering questions from that post are:

(1) Should the responses to irrelevant questions NULLed out?
(b) What is the preferred way to deleting the value from $_SESSION[SGQA]? unset($_SESSION[SGQA])? $_SESSION[SGQA]=''? (I'll want to delete those values so that future equations don't use them if those questions are irrelevant)

(2) Relevance could let a survey finish quickly - e.g. don't ask the last 10 groups of questions if the subject is a minor. However, in such cases, the subject will not see the Submit button since that is only present on the last Group.
(a) Is that behavior acceptable, or should the system try to detect that all future groups would be irrelevant and in such cases make the Submit Button visible?


lemeur (developer)

Last edited: 2011-07-14 22:21

View 2 revisions

1) => yes if deletenonvalues=true.
1b) Good question, I think this was never addressed before because I haven't implemented chained conditions. So there is no current good way of doing this. However I think that just unsetting the SESSIOn[SGQA] might have side effects. Currently, Nulling the value is only done in the SQL query, not in SESSION.

2) True, this is also an issue in current Code. In the past there was always a very Last page saying "thanks, you can now confirm your answers", but participants used to forget clicking this last submit button (resulting in incomplete answer in DB). We changed that so that this last page is no more displayed, hitting the NEXT button in a group that is the last to be displayed (because others are hidden) will result in an automatic and transparent Submit (even if the last button pressed wasn't labelled as a "Submit").
2a) It would be perfect to implement this, but since conditions can used answers from the current page, it would have to be compueted online (Javascript eval).


lemeur (developer)

About point (2a), I wonder if this wouldn't be too much CPU demanding for the client. Indeed, we have seen this issue in surveys havinf a lot of questions and conditions. Having conditions on "Groups" rather than only on questions could help fix this problem though.


TMSWhite (reporter)

I haven't tried (2a) before. However, since Relevance is Question Attribute, I can retrieve all of them for the survey in a single query. All of the current values only need to be loaded once too. The unit tests take less than a second to perform, so I'd expect that even with 2000 questions, it would still take less than a tenth of a second to evaluate all of subsequent relevance equations.

However, I agree that attaching relevance to Groups would be nice.


TMSWhite (reporter)

Status Update: Good News - the hard part is done and fully unit tested.

All Expressions that are valid server-side are also valid dynamically (e.g. via JavaScript) as long as all functions used in the Expression have JavaScript equivalents.

Currently all operators and most math function (but not math contants like PI and E) work.

It will be possible to add missing JavaScript functions - e.g. in an ExpressionManager.js file.

The next steps are to:
(1) Ensure that all variables used in Expressions on page have their <input type='hidden' value'xxx'/> nodes written to the page
(a) ExpressionManager does collect the list of all needed JavaScript SGQA names for the page
(b) So need to figure out where LimeSurvey does the equivalent so that I don't write the same hidden input values twice
(2) Create JavaScript functions for each question that use the JavaScript-equivalent of the Relevance Equations, and have them control the question's visibility
(a) Naming: ExprMgr_relevance_Q - where Q is the question number
(b) So need to figure out whether should call that sequence of functions from both noop_check_conditions() and check_conditions()
(c) May want to call from both so that don't have to modify all other <input> fields to know which of those two functions to call
(d) Probably have both noop_ and check_conditions() call ExpressionManager_update_dynamic_values() which would call all relevance() equations in sequence (so get chaining behavior)
(3) Integration dynamic functions into conditional tailoring - so if text is dynamic on the page (like showing running totals of Assessments, or change text of questions based upon gender or number of subjects), that text will also update automatically.
(a) Naming: ExprMgr_tailor_SGQA_X where X is sequential number of ExpressionManager replacements in the string. So, if there are 4 replacements, you'd have ExprMgr_tailor_1X2X3_1() - ExprMgr_tailor_1X2X3_4() and <span> elements with the same ID. That way all of the tailoring can be called dynamically from ExpressionManager_update_dynamic_values()


lemeur (developer)

Hi Tom,

For token-attributes, the values are added directly by while generating the javascript evaluation code.

For previous questions' answer, when in group-by-group mode, the hidden fields are added in group.php line 1365:
        //if conditions exist, create hidden inputs for 'previously' answered questions
        // Note that due to move 'back' possibility, there may be answers from next pages
        // However we make sure that no answer from this page are inserted here


lemeur (developer)

2b) I'm a little confused here. noop_checkconditions is an empty javascript function only used when the question isn't used in a condition (value $ia[8] coming from _SESSION['insertarray']). But maybe the issue here is that the ExpressionManager is used for a wide range of features, and not only conditionnal question branching. Is that your problem ?

Or is it connected to the chaining conditions issue ?

Can you enlighten me on this issue ?


TMSWhite (reporter)


You are essentially correct.

Despite what I proposed in the initial issue post, I'm not going to use the $ia[] array at all. Moreover, ExpressionManager can be used in many places, so the current logic for deciding whether to use noop_checkconditions() vs. checkconditions() won't work as is.

Here's what I'm doing instead:
(1) Relevance is stored as a Question Attribute (instead of in $ia)
(2) ExpressionManager will be used, at least, for processing Relevance, Questions, Answers, and Templates. Once that is working, I'll also have it process Validation too
(3) To determine which variables are used on a page, I'm making it do this: For every question in the active group:
(a) Evaluate the Relevance Question Attribute, and collect the list of variables used
(b) Evaluate the Question and Answer string (anything that could have replacement values), and collect the list of variables used by them
(c) create a unique list of variables used by anything that uses ExpressionManager on that page, with attributes saying whether or not they are set on the current page.
(d) Create <input type="hidden"> nodes for the subset of (1c) that has not already been created by the existing Conditions code (group.php:1365)
(e) Create ExprMgr_relevance_Q() functions for each question that has a Relevance Attribute - this will control the question's visibility
(f) Create a <span id="ExprMgr_tailor_SGQA_X"/> and ExprMgr_tailor_SGQA_X() function for each tailored component of a String (e.g. Question or Answer) where any of the tailored content (like INSERTANS:SGQA) uses one of the variables from (1c)
(g) Create a function ExpressionManager_update_dynamic_values() which will be called by both noop_checkconditions() and checkconditions()
(h) Once we're ready to retire Conditions, we can return to the idea of a noop and real call to ExpressionManager_update_dynamic_values(). (1c) will know which variables might require an change to the DOM model, so we could use a noop_ExpressionManager() call for the other variables on the page.
(i) To avoid too many changes to the page each time a value changes, I'm going to compare the newly generated values to the existing ones and only call document.getElementById().value=newvalue when the value has really changed.


TMSWhite (reporter)

Status Update: The JavaScript needed to do dynamic relevance and tailoring now works for the stand-alone test cases.

This can be tested at http://localhost/limesurvey_dev_tms/classes/eval/Test_ExpressionManager_Relevance.php

Next step is to finish integration so that this controls operational surveys.


TMSWhite (reporter)

Initial integration into LimeSurvey 1.91 (limesurvey_dev_tms) branch completed.

As of revision 10575, Questions can be dynamically be made visible based upon Relevance, and dynamic tailoring works.

A few minor bugs to work out and it will be done:
(1) Dynamic features don't work on first page until go past it and come back
(2) Need to figure out right way to deal with caching (esp for switching languages)

This also highlights the need for
(1) Total re-design of logic in survey.php and group.php to simplify it (e.g. remove processing of conditions and assessments - this should be a focus for 1.92 / CI
(2) Better control of focus() - when a new question appears it is not getting focus.

Once this is fully working, I'll upload a sample survey so people can see how this works.


TMSWhite (reporter)

This now works in revision 10579, with versions of tailoring and relevance calculations using much-improved in Javascript.syntax highlighting and tool-tipping.

It also supports real-time changes to question visibility and micro-tailoring on the current page without visibility.


lemeur (developer)

Hi Tom,

I have fixed the first page issue for both tailoring and branching logic.
However, the "Releavance" question attribute is filtered when using the GUI (closing '}' and single quotes are stripped off).


TMSWhite (reporter)


Thanks - hadn't tried radio buttons yet.

Here's a good sample survey to try (attached).



TMSWhite (reporter)

Attached is an updated demo (same filename, however) that demonstrates:

(1)Calculations, Assessments & Reports - Page 1 shows many math facts for the numbers as you enter them
(2)Conditional Relevance (e.g. complex Conditions) with cascading - Page 2 conditionally asks questions about your age, marital status, and number of children, but only showing potentially relevant questions based upon your age and prior responses
(3)Conditional Micro-Tailoring - Page 2 provides a running commentary about the information you entered at the bottom of the page
(4)Blanking out Non-Applicable (e.g. irrelevant) responses. Page 2 shows that if you first enter an adult age and information about children, then change the entered age to 4, all of the questions that depend upon age get blanked out. This feature is used to support cascading logic in the Relevance equations so that you don't have to attach all relevance criteria to each question.


TMSWhite (reporter)

Dynamic relevance and tailoring now works for survey.php. Just take the sample survey and change it to use survey.php to see how it works

This means it should also work for dataentry.php, but I haven't looked into that.

A new lingering issue (which I've added to the ExpressionManager TODO list on the wiki) is to get the focus/tab order correct for questions that dynamically appear. This applies to both survey.php and group.php integration.


TMSWhite (reporter)

Finished testing ability to hide/show each question type using new demo (attached).


TMSWhite (reporter)

Note to self.

Seems like only realistic way to NULL out responses on a question-by-question basis (e.g. like what a RESET button would do on a standard HTML page, but one question at a time) is to add a reset_question_qid() JavaScript function for each question on the page. This would be generated within the do_xxxx() functions of qanda.php (like do_5pointchoice()), since those functions know the names of all of the answerSGQA variables for checkboxes, lists, etc. Then, ExpressionManager would call reset_question_qid() whenever the question is irrelevant.

This would guarantee that checkboxes are unchecked, list items are unselected, rankings are reset, etc.


lemeur (developer)

Reseting a question answer to NULL at runtime has been disabled in current LS1 version because it bypasses the Default Values settings. Instead we decided to set hidden answers to NULL in DB only and at submit-time only.

If you decide to NULLify the answers at runtime, take care to respect Default values that could be set for hidden questions.


lemeur (developer)

Last edited: 2011-08-01 10:15

View 3 revisions

Humm, Correction of my last post.
The current LS1 approach of Resetting answers only at submit time ok as long as we do not support chained conditions.

Attached is a Demo Survey showing the bug (test-tibo1.lss):
* Q1: answer Yes ==> Q2 is displayed
* Q2: answer C ==> Q3 is displayed
* Q3: answer 10, and then NEXT to go to next group
* Q4 is displayed because Q3 is not 5, go Back to previous group
* Q1: now change your answer to N ==> Q2 is hidden BUT Q3 is still displayed.

With support for chained conditions, we need to make sure that hidden question answers are not used by either resetting the answer to a Non applicable value OR by ignoring such answers.

If we decide to reset answers to Non applicable value, then we need to set the default value if it is set as soon as the question is displayed again!!

To sum things up, 2 solutions possible.

Solution 1:
* we need to reset the hidden question answers to a non applicable value
* we need to set the default value of each answer by using dynamic javascript instead of using PHP while generating the web page

Or Solution 2:
* we keep record of the displayed status of each question, and EM checks if the status of the question is "displayed" before trying to use the answer value. If the question is now hidden, let's ignore the value of the field
* when the question is displayed again, let's forget about the default answer and let the previously entered answer displayed (because the participant has already seent he default value, and has decided to change it in the past).

Solution2 is easier and yet remain logical for the participant. I vote for it.


TMSWhite (reporter)

Interesting. I was working towards Solution 1 using JavaScript. I hadn't gotten to the default values yet - I was just making sure I could both NULL the stored value and also reset the GUI - e.g. uncheck radio buttons (or set them to NoAnswer), unselect list boxes, etc.

Although Solution 2 sounds like it should be easier, it might not be. People may do conditional logic based upon the default value. Shouldn't such logic work even if the question is hidden? If so, we'd need to do Solution 1 anyway.

We you voting for Solution 2 because Solution 1 seemed too hard (it's not), or because you think Solution 2 is better?


lemeur (developer)

I was voting for solution 2 because:
a- I think it will impose a lot of Javascript on the client-side. We had a lot of issues with client-side performances when using a lot of conditions, so I would prefer limitting the Javascript CPU footprint to the strict minimum
b- I really think that when you change your mind twice, you would be happy to find your previous answers and not the default values (but this is a personnal preference)
c- and yes, I still think solution 2 is easier to implement because it requires very little mods (the displayXXX HTML element exists and we can keep trak of it), no need to change the do_XXXX in qanda.php, ... but this point is to be decided by the brave man who will implement this ;-)

I would be happy to have Carsten's opinion on this.
Carsten are you listenning to this thread ?
If not, Tom can you ask the question to him ?


lemeur (developer)

[QUOTE]People may do conditional logic based upon the default value. Shouldn't such logic work even if the question is hidden?[/QUOTE]

I don't think so. If the question is hidden, then it is not relevant. This is also the case for any default answer: not relevant.


TMSWhite (reporter)

OK, I'll give Option 2 a shot and see whether the logic works as expected. Options include:
(1) If any of the variables are Not Applicable, then the whole expression is FALSE
(2) For each non-applicable variable, return an empty string.

Advantages to (2) include having sum(), implode(), etc. work correctly. I think it should also make &&, ||, etc. work as people would expect, but I'll test this.

What about for sub-questions? I haven't played with that functionality, but it sounds as though there is a way to selectively show/hide subquestions within an array. Presumably that would make just those sub-questions irrelevant.

How do I detect that sub-questions are hidden?


lemeur (developer)

Last edited: 2011-08-02 11:00

View 2 revisions

Damnit, completely forgot about array_filter... arghh... I don't like this feature because it is an add-on that is evaluated side by side with conditions.
In my opinion it would be better to have the array_filter code rewritten so that it computes an EM expression from the array_filter attribute.
Humm.. this may require changing the checkmandatory() && checkconditionalmandatorys()code...

This doesn't answer your question though. I'm not sure if there is a displayXXX element for subquestions, but it would seem logical to have such element.


lemeur (developer)

Humm.. anyway, in current EM version, there is a bug when a hidden question is mandatory.
Check the uploaded test-tibo2.lss:
* you can answer (or not the 1st and second question [except option C for the 2nd question) so that question 3 is not displayed
* try to go to next page (Q" not displayed is mandatory ==> you can't go to next page)


TMSWhite (reporter)

Can you attach an example of using array_filter? I've never used that functionality.

For checkmandatory(), we may went to re-do that anyway. When I play with the survey I uploaded, and you don't answer some irrelevant questions the first time, then back-up and make them relevant, they all appear in red since they weren't answered the first time. That looks odd and may not be the desired behavior.

Additionally, I've used EM to have relevance "trump" mandatory so you can use relevance to hide mandatory questions yet still be able to go on to the next page. If the question is shown, then mandatory applies, but if it is hidden, mandatory is ignored. Is that the desired behavior?


TMSWhite (reporter)

I implemented Option 2 from note 15914. Cascading conditions (relevance) is working as expected in ExpressionManager-Demo-AllQuestionTypes.lss

Now, every time EM tries to access a variable, it calls ExprMgr_value(jsName,displayNum), where jsName is the JavaScript name for the hidden input item that stores the value, and dislayNum is the question Number. EM returns '' if the question was not displayed.

Once we know the proper naming convention for naming sub-questions, we can extend that function to ExprMgr_value(jsName,displayNum,subQuestionId) so that we can test both that the question was shown and also that the sub-question was shown.


lemeur (developer)

[QUOTE]If the question is shown, then mandatory applies, but if it is hidden, mandatory is ignored. Is that the desired behavior?[/QUOTE]
Yes it is the desired behaviour.

[QUOTE]EM returns '' if the question was not displayed.[/QUOTE]
I feel that the emtpy string is not a good idea.
Imagine that I set a condition on Question3 if Question2 (a text question) received no answer, and I add a dynamic text in Question4 "You gave no answer to Question2, can you explain why?").
If you're unsing the empty value for questions hidden and if Question2 is hidden, then this example would not work because Question4 will ask the participant for a reason why he did not answer a hidden question !!!

In fact we need a specific Not-Applicable value (such as NULL), ans force EM to return "False" (which I guess is the default when the expression can't be evaluated).
What do you think ?

[QUOTE]Can you attach an example of using array_filter?[/QUOTE]
yep, will do ASAP. The principle is to use Question1 as a "multiple choice" with Question title "Q1" and answers A, B, C, then add a Question2 or type "Array XXX" with subquestions using codes A, B, C. Then on Question2 add the question attribute array_filter and set it to "Q1".


lemeur (developer)

test survey for array_filter added


lemeur (developer)

[QUOTE]I implemented Option 2 from note 15914. Cascading conditions (relevance) is working as expected in ExpressionManager-Demo-AllQuestionTypes.lss[/QUOTE]

test-tibo1.lss (described in post 15914) now works.


TMSWhite (reporter)

For test-tibo1.lss, what is the desired behavior for Q4 in G2? It depends on the value for Q3. If Q3 is not relevant, should Q4 be displayed?


TMSWhite (reporter)

I expect that the behavior you want to Q4 is that it should not be shown when Q3 is irrelevant (not shown).

Is there an existing way to know which questions were hidden on other pages? If not, I can create an array mapping question# to hidden status and check that when generating the JavaScript.


TMSWhite (reporter)

I think the current solution may already do what you are hoping for(except for the off-page hidden status mentioned in post 15950).

There are several examples in ExpressionManager-Demo-AllQuestionTypes.lss that provide tailored messages based upon whether the person answered certain questions (on page 2 of the survey). Each uses the is_empty() to check whether there is a response. For the expression is_empty(age) will return true if age was shown but not answered, but will return false if age was not shown (regardless of whether it was answered).

So, I think as long as I can deal with the the off-page hidden status, this should address the concern in post 15945 without needing special values for NULL.

FYI, the reason for returning '' is that Boolean('')===false, but Boolean('false')===true. So, returning 'false' or 'NULL' will make the Boolean logic fail.


TMSWhite (reporter)

About post (15924), yes, array_filter might need to be rewritten to be called by ExpressionManager. That way, the dynamic show/hide of questions will only occur if the question is relevant. If we don't re-write array_filter, there is a small risk that the rows might appear even if the question is hidden. I tested this on test-array-filter.lss, and if I make Q2's relevance=0, the question does stay hidden in Firefox even as options are changed in Q1. I don't have an easy way to test whether this works for all browsers.

The logic for array_filter seems pretty easy, so if we do need to re-write it, it shouldn't be too much work.


TMSWhite (reporter)

OK, I've implemented off-page relevance checks, and I'm still now sure what the desired behavior should be for test-tibio1.lss

The relevance for Q4 is currently (Q3!=5).
If Q3 is set to 10, then hidden, the relevance resolves to ('' != 5), which is true, so Q4 is displayed. That seems reasonable, but you may have expected cascading relevance.
On the other hand, if Q3 is set to 5, then hidden, the relevance also resolves to ('' != 5) which is true, so Q4 is also displayed. That behavior seems appropriate.

If you want Q4 hidden if Q3 is hidden, there are two obvious options:
(1) Change the relevance equation to (!is_empty(Q3) && (Q3!=5))
(2) Have relevance return false if any of the source variables are hidden.

The problem with #2 is that you might have a relevance like:
(sum(Q1,Q2,Q3,...Q10)> 5)) - in that case, you would not want it to always be false if any of those variable are hidden.

So, option (1) seems like the way to go, but we could add some other helper functions to make it easier to do cascading. Perhaps:
(1) if_shown(a,b,c,d,e,...) - returns true if and only if all of those variables are relevant

Regardless, this is now working in JavaScript. Now I have to go back and get the PHP working too (e.g. have it also check the relevance status of the source variables when retrieving their values).


lemeur (developer)

Expected behaviour for Q4: if Q3 is hidden then the condition Q3!=5 can't be evaluated and thus Q4 is hidden.

What we need is a macro such as the shell bash has:
-bash-4.0$ vech=Bus
-bash-4.0$ echo $vech
-bash-4.0$ echo ${vech:-Car}
-bash-4.0$ unset vech
-bash-4.0$ echo ${vech:-Car}

this way the SUM({Q1:0},[Q2:0}, ...) can be evaluated even if one question is hidden.

What do you think?


lemeur (developer)

In case we had wanted Q4 to be displayed because Q3 (hidden) isn't equal to 5, then we should have written a condition like:
Q3!=5 OR !is_shown(Q3)

This means that writing Q3!=5 really means: is_shown(Q3) && q3!=5

I think this is the best way to be consistent.

What do you think ?


lemeur (developer)

from post 15950: "Is there an existing way to know which questions were hidden on other pages?"

I don't know if the displayXXX elements of previous questions are written to the following pages.


TMSWhite (reporter)

Hmmm. We may want to let others weigh in too.

In the most recent release, I implemented option (1) from post 15968. That's the reverse of what you're proposing.

How about having a global variable within the survey so that users can decide the default behavior when an equation contains irrelevant attributes. If they want to force the expression to return false ('') if any of the contributing variables are irrelevant, then the system will add an 'is_shown(...) && ' clause to the beginning of their expression, like this:

$expression = // value for expression from the database
if (Force_Expression_False_If_Any_Variables_Irrelevant) {
  $expression = 'is_shown(' . implode(',',variables_used) . ') && '. $expression;

The other tricky issue is that this only applies to the current page. If we try to access an irrelevant variable from a prior page, it will have been NULLed out by the time we get to the current page anyway. At least that is what is currently happening. If I should be keeping the value in $_SESSION[] even though it is NULL in the database, please let me know.


TMSWhite (reporter)

from post 15971.

I ended up creating hidden variables with id='relevanceNNN' where NNN is the question number. Relevance and Display are somewhat independent - you could have an always hidden question (like an equation, or a pre-filled response) that is still relevant. However, if it is not relevant, it can't be shown.


c_schmitz (administrator)

Last edited: 2011-08-04 10:11

View 2 revisions

Although it is more logical to keep relevance and display apart, it will certainly be above the understanding of the average user.

Users built cascading conditions all them time and are expecting that when e.g. Q3 was not shown any other questions having conditions on Q3 were not shown either (and so they always evaluated false). We should keep this behaviour.

Having an always hidden question is IMHO not part of the expression manager but only a special case and part of display management, so it shold be not considered. They should be treated like a normal questions with a normal visible response.

In addition there are two forms of hidden: visually hidden in the page (that is needed if you have custom javascript that relies on a hidden value) or completely hidden (=no output at all), which might be needed for values that are passed through the survey by URL. Anyway, in your scenario these 'hidden' fields should always be considered as relevant.


TMSWhite (reporter)


This gets tricky. I'll create a wiki-page section to discuss further. They underlying questions are:
(1) Which operators should always return false if any of their arguments are irrelevant?
(a) Comparators? (==, !=, >, >=, <, <=)
(b) Math? (+, -, *, /, +=, -=, *=, /=)
(c) Logic (&&, ||, !)
(2) Can we safely apply the above to just plain variables (e.g. a == b), or do people truly want this to apply to sub-equations (e.g. (a+b)==c). That latter is much harder.
(3) Is it always safe to NOT return false for functions that take multiple arguments - like sum(a,b,c), if(a,b,c), list(a,b,c), max(a,b,c) - e.g. each sub-equation (the parts between commas in the function call) will be false, but an argument will be passed to the function.


TMSWhite (reporter)


I'm not sure what you mean by [quote]Having an always hidden question is IMHO not part of the expression manager but only a special case and part of display management, so it shold be not considered. They should be treated like a normal questions with a normal visible response.[/quote] Can you clarify? If you are saying that the Expression Manager (EM) should only consider relevance status, and not display status, then I agree. For example, the Equation question type is usually hidden (except during development/debugging), but as long as it is relevant (even if hidden), it should be processed by EM normally.

Also, for the forms of hidden:
(1) Question Attribute 'hidden' (for Always hide this question) - these are always relevant unless the author uses a relevance equation for that question
(2) Hidden by JavaScript calls - if someone calls $('#displayNNN').hide(), I'd have no way of knowing, so relevance status would stay unchanged
(3) Values passed through from the URL - Please provide an example survey that uses this - I don't have experience with PASSTHROUGH arguments yet, and don't fully understand how it is supposed to work.


lemeur (developer)

I agree that EM should only consider relevance and not the display status.
When Carsten talks about an "always hidedn" question, he's refering to the 'hidden' question attribute which is sometimes used to have a field created in DB. Then, if a participant accesses the survey with a specific GET parameter having the same name as the DB field of this question, then the corresponding value is inserted into the repsonse table.

For instance, the hidden question has DB field 1234X1X2, and the participant uses a URL such as http://surveyhost/limesurvey/index.php?sid=1234&1234X1X2=Y&...

About the points you raised:
1) Exactly
2) Then we can't know, let's forget about these cases. I think that EM will make most of these tweaks obsoleted
3) see my example above


TMSWhite (reporter)

The desired cascading relevance functionality is available in the 10722 revision. has been updated to show the strategy taken.

The Refactoring Update sections of provide a little more detail.

It will still be possible to do the Q1:0 syntax (post 15969) if needed - this would internally be treated like Q1.NAOK, but with a different default value (the current default for NA is 0)


TMSWhite (reporter)

This is available in >= 2.0alpha

-Issue History
Date Modified Username Field Change
2011-06-12 10:54 TMSWhite New Issue
2011-06-12 18:16 lemeur Note Added: 15415
2011-06-19 07:07 TMSWhite Note Added: 15501
2011-06-19 11:09 lemeur Note Added: 15503
2011-06-19 17:57 TMSWhite Note Added: 15508
2011-06-19 18:59 TMSWhite Relationship added related to 05288
2011-06-19 19:01 TMSWhite Relationship added related to 05103
2011-06-19 19:01 TMSWhite Relationship added related to 05268
2011-06-19 22:10 lemeur Note Added: 15513
2011-06-19 22:18 lemeur Note Added: 15514
2011-06-19 22:31 TMSWhite Note Added: 15515
2011-06-20 03:48 TMSWhite Note Added: 15516
2011-07-14 18:54 TMSWhite Note Added: 15769
2011-07-14 21:15 lemeur Note Added: 15770
2011-07-14 21:24 TMSWhite Note Added: 15771
2011-07-14 22:20 lemeur Note Added: 15772
2011-07-14 22:21 lemeur Note Edited: 15772 View Revisions
2011-07-15 09:56 lemeur Note Added: 15775
2011-07-15 12:52 TMSWhite Note Added: 15776
2011-07-19 00:19 TMSWhite Note Added: 15790
2011-07-19 09:31 lemeur Note Added: 15793
2011-07-19 09:44 lemeur Note Added: 15794
2011-07-19 16:12 TMSWhite Note Added: 15798
2011-07-21 23:14 TMSWhite Note Added: 15822
2011-07-23 09:09 TMSWhite Note Added: 15845
2011-07-24 07:31 TMSWhite Note Added: 15848
2011-07-24 10:50 lemeur Note Added: 15850
2011-07-24 13:09 TMSWhite Note Added: 15851
2011-07-24 13:10 TMSWhite File Added: ExpressionManager-Demo.lss
2011-07-26 20:15 TMSWhite Note Added: 15871
2011-07-26 20:15 TMSWhite File Deleted: ExpressionManager-Demo.lss
2011-07-26 20:15 TMSWhite File Added: ExpressionManager-Demo.lss
2011-07-29 15:36 TMSWhite Note Added: 15886
2011-07-30 22:13 TMSWhite Note Added: 15911
2011-07-30 22:13 TMSWhite File Added: ExpressionManager-Demo-AllQuestionTypes.lss
2011-07-31 14:43 TMSWhite Note Added: 15912
2011-08-01 08:58 lemeur Note Added: 15913
2011-08-01 10:12 lemeur Note Added: 15914
2011-08-01 10:13 lemeur Note Edited: 15914 View Revisions
2011-08-01 10:14 lemeur File Added: test-tibo1.lss
2011-08-01 10:15 lemeur Note Edited: 15914 View Revisions
2011-08-01 16:02 TMSWhite Note Added: 15917
2011-08-01 20:42 lemeur Note Added: 15920
2011-08-01 21:53 lemeur Note Added: 15922
2011-08-01 23:49 TMSWhite Note Added: 15923
2011-08-02 10:55 lemeur Note Added: 15924
2011-08-02 11:00 lemeur Note Edited: 15924 View Revisions
2011-08-02 11:05 lemeur Note Added: 15927
2011-08-02 11:06 lemeur File Added: test-tibo2.lss
2011-08-02 14:44 TMSWhite Note Added: 15931
2011-08-02 19:15 TMSWhite Note Added: 15944
2011-08-02 19:38 lemeur Note Added: 15945
2011-08-02 19:44 lemeur File Added: test-tibo-array_filter.lss
2011-08-02 19:44 lemeur Note Added: 15946
2011-08-02 20:22 lemeur Note Added: 15948
2011-08-02 20:29 TMSWhite Note Added: 15949
2011-08-02 20:39 TMSWhite Note Added: 15950
2011-08-02 20:44 TMSWhite Note Added: 15951
2011-08-02 21:10 TMSWhite Note Added: 15953
2011-08-03 20:16 TMSWhite Note Added: 15968
2011-08-03 22:22 lemeur Note Added: 15969
2011-08-03 22:25 lemeur Note Added: 15970
2011-08-03 22:29 lemeur Note Added: 15971
2011-08-03 22:54 TMSWhite Note Added: 15972
2011-08-03 22:57 TMSWhite Note Added: 15973
2011-08-04 10:01 c_schmitz Note Added: 15975
2011-08-04 10:11 c_schmitz Note Edited: 15975 View Revisions
2011-08-04 13:52 TMSWhite Note Added: 15978
2011-08-04 16:42 TMSWhite Note Added: 15979
2011-08-04 17:44 lemeur Note Added: 15980
2011-08-14 11:40 TMSWhite Note Added: 16057
2011-10-28 23:05 TMSWhite Note Added: 16567
2011-10-28 23:05 TMSWhite Status new => resolved
2011-10-28 23:05 TMSWhite Fixed in Version => 2.00
2011-10-28 23:05 TMSWhite Resolution open => fixed
2011-10-28 23:05 TMSWhite Assigned To => TMSWhite
2012-06-21 13:22 c_schmitz Status resolved => closed
+Issue History