Anonymous Login
2017-01-22 19:22 CET

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
05279Development Survey Designpublic2013-01-06 01:38
Assigned To 
Product Version1.90+ 
Target VersionFixed in Version 
Summary05279: Add a GUI for ExpressionManager
DescriptionNow that ExpressionManager may be used to manage all replacement of things within curly braces (e.g. TOKEN:, INSERTANS:, LimeReplacementValues, Macros used by the template engine, and expressions), it may need a GUI.

The current GUI for the non-expression replacements will continue to work fine even if ExpressionManager takes over the run-time resolution of those values. So, the focus of this issue is just on helping users build valid equations.

I am not an expert in creating GUIs, so someone else would need to take lead on this. However, ExpressionManager has features that should help enable GUI creation:
(1) When there are errors, it reports the exact offset in the string where those errors occur. So, we might envision a pop-up window that shows the source String, and for each error, shows the offset in the string via a ^, with a message underneath. There are probably even prettier ways to do this, but at least ExpressionManager will help identify where the problem occurs.
(2) Although the errors are currently in English, once we're happy with their wording, we could use $clang->gT() to provide translations for those values.
(3) ExpressionManager also reports balanced parentheses (e.g. too few or too many parentheses), and their offset in the string
(4) It reports whether there are variables that it doesn't recognize
(5) It reports when there are functions it doesn't recognize, or cases of passing the wrong number of arguments to a function
TagsNo tags attached.
Attached Files

related to 05103closedTMSWhite User patches Support conditional piping/tailoring and complex calculations via embedded equation parser 
related to 05268closedc_schmitz User patches Do all LimeReplacementField and Token replacements in a single function 
has duplicate 05503closedlemeur Development  Discussing the future conditions GUI for EM 



TMSWhite (reporter)

Upgraded ExpressionManager error reporting to now show the problem equation, but formatted such that:
(1) Whole equation is highlighted
(2) Any known problems with syntax are red-boxed (e.g. to show unknown variables or extra closing parentheses)
(3) The equation is ToolTipped to show known errors

See Issue 05268 for patch demonstrating that feature


TMSWhite (reporter)

On the front-end side, perhaps we could do incremental search-based code insertion.

So, especially for large surveys, instead of picking a question or answer from a drop-down box, the user could optionally type the first few characters of the variable name or MACRO name, and a tool-tip will show the possible values (like Google's incremental search). The ExpressionManager knows the set of available variable names, MACRO names, and functions, so if someone knows how to too the ToolTip, they'd have the data they need to support it.

Then, if the user selects a function, we could try to show function-related help similar to how IDEs (like NetBeans) do it - showing the # and name of required/optional parameters, plus the purposes of the function.

ExpressionManager already knows the number of required parameters for the functions, plus a short description of the function, so we'd just need to add some extra fields to its array showing the allowable function syntax for each number of variables [that is what we'd insert at the cursor if the person selects the function], plus some narrative description that could be easily translated.


lemeur (developer)

about using "$clang->gT()" you should use it right from the beginning because if no translation exist it displays the original text. Then, if we want to change the wording we can do so by either replacing the wording in the code, or replacing the wording in the English2English translation file (though it could become confusing for non-english translator I confess).

About post 15475: very interresting idea.

About post 15485: Ouahh, that would be wonderful, but also a great deal of work. I wonder if a plugin inside CKeditor could do that.


lemeur (developer)

Last edited: 2011-07-24 17:34

View 2 revisions

As written in a thread on the forum (MIT licence) would do the trick: syntax coloring, and auto-complete. Simply modifying the file javascrip.js which is the parser for JavaScript code would be ok.


TMSWhite (reporter)

Notes to self:

(1) Add GetPrettyPrintString($eqn)
(a) split $eqn into Strings and Equations, color-code String black (regular text)
(b) Tokenize and validate Equations
(c) <span> to highlight full equation in light yellow
(d) <span> to red-box and tool-tip errors (as is done now)
(e) newly color code the following:
 - valid punctuation (parentheses, operators, gt/lt/eq/etc.)
 - function names -
 - strings - (distinguish single vs. double quote?)
 - constants -
 - variables:
   -- read-only, defined on current page -
   -- read-only, defined on another page -
   -- read/write, defined on current page -
   -- read/write, defined on another page -
 - attributes (eq. q1.code vs. q1.value)?

(2) Update way variables are passed to ExpressionManager
Instead of having Variables vs. NamedConstants, also need to know which variables are defined on the current page, so, pass instead an array of:


(3) Add ValidateEquation($eqn,$varlist)
(a) To be called when set Relevance Question Attribute and/or putting equations into Questions, Answers, etc.
(b) passed a list of $varList customized to the current group, tokenize test the $eqn for syntax and compile errors (this will validate that the equation isn't trying to access variables that haven't yet been set)
(c) if any errors, re-diplay the question showing the GetPrettyPrintString()
(d) Modify Question display to always show Relevance if it is non-null (similar to showing conditions for a question and/or questions that depend upon it).
(e) Also use this to color code and validate equations for Print view of survey (e.g. showing all questions and relevance conditions on a single page) - this is way to validate logic of complete survey.

(4) Add new Testing page linked to http://localhost/limesurvey_dev_tms/classes/eval/ExpressionManagerTestSuite.php
(a) Let user specify variables using syntax in #2 - pull default values from unit tests
(b) Let user specify list of equations to test - pull from default values in unit tests. Syntax is ExpectedAnswer~Equation
(c) Generate color-coded results page similar to http://localhost/limesurvey_dev_tms/classes/eval/Test_ExpressionManager_Evaluate.php

(5) Add GetJavaScriptEquation($jsResultVarName, $eqn,$varlist)
(a) used to generate JavaScript equation on the pages
(b) If $eqn does not depend upon or set any variables from the current page, return null (in such case, than call EvaluateString() to compute the value of the equation and return the result)
(c) Otherwise, convert the source $eqn into a JavaScript function that returns the result of evaluating $eqn and sets the value of $jsResultVarName.
 - When setting the result, it should update the .value() in the DOM object so the the value dynamically changes
 - This should let us change the values of conditionally tailored strings too - e.g. if conjugating verbs and declining nouns withing questions or answers (as long as they are in their own <span>)
 - Ideally, condense the $eqn - e.g. pre-process all portions of the equation that do not use dynamic values that might be set on the current page
(d) Add column to Test_ExpressionManager_Evaluate.php to show generated JavaScript code


lemeur (developer)

Was the last post really targetting this exact ticket ? Not sure, but this is no problem.

About the GUI, I find usefull to be able to reuse conditions set on a previous question to new questions. Copy-condition was helpfull in this way: we could even only copy a subpart of a condition to other conditions. This will be a little more tricky with the expression manager.

For instance, one question gets the condition (Sex = Male and Country = France) or (Sec = Male and Country = Germany) And Status = Married And ......
Then another question gets the condition: (Sex = Male and Country = France) or (Sec = Male and Country = Germany) And Status = Single And ......
It would be great to be able to only reuse a subpart of the condition.

So I'm thinking about something like:
1- register a condition/variable as {ScenarioA} = (Sex = Male and Country = France) or (Sec = Male and Country = Germany)
2- be able to set conditions such as: {ScenarioA} And Status = Single And ......

Does this make sense to you Tom ?


TMSWhite (reporter)


My last post started on topic, but ended up hitting a mix of ExpressionManager needs. Basically, those are needed as infrastructure to (a) provide good user feedback for the LimeExpert mode, and (b) provide the core methods that the GUI would eventually need.

Regardless, yes, the way I deal with scenarios is via the Equation question type. You'd create a question called ScenarioA with text {Sex == 'Male and (Country == 'France' or Country == 'Germany')}. Then, you could use Scenario A in a future Relevance Equation (e.g. {ScenarioA and Status=='Single'})


lemeur (developer)

Hi Tom,

I was just kidding about the irrelevant post, and I agree with all the points you've described above.

The way you deal with scenario is OK for me, however I think the real syntax would be {INSERTANS:xxxx and Status=='Single'} with INSERTANS:xxx pointing to the ScenarioA question: am I right ?

If yes, then this makes reading the expression a little more difficult. In general this reminds me of a question you asked to Carsten about the question Code beeing unique or not inside a survey. The answer was that 'nothing' enforces it to be. I don't know if there was another reason given for this apart from "this is done this way for now". If this code was unique inside the survey, it would be very easy to have INSERTANS:QuestionCode which would be far easier to read IMHO. I think it's worth speaking about this again with Carsten.
What do you think ?


TMSWhite (reporter)


ExpressionManager can provide access to several different question attributes, and several aliases for the question. Say I have a question with title='Country' located at SGQA code 1X2X3, you'd be able to use the following:
(1) INSERTANS:1X2X3 -- the display value of the response
(2) Country -- the coded value of the response
(3) Country.shown -- the display value of the response
(4) Country.question -- the text of the question asked

We can easily extend ExpressionManager to provide access to other question, group, or answer attributes.

So, yes, the syntax would be what I showed above. You can continue to use SGQA codes if you want, but you can use Question Title.

Although Question Title is not officially unique, any user who exports to SPSS or SAS will need to make sure it is unique, since SPSS generates a
VARIABLE_LABEL=xxxx line, where xxxx is Question Title. So, if it isn't unique, SPSS will throw an error.

So, I think we should give users the option of checking whether Question.Title is unique.



lemeur (developer)

Ah ok,
That's very great, and I agree about letting our users decide if they will enforce or not unique Question-titles (what I usually call 'code' by mistake).
I even vote for having this as the default value.

(3) Have you already implemented this ? If yes, that very useful.
(5) It would be also great to have a 'Country.answered' in addition to Country.shown. Indeed, sometimes we need to know when a question is displayed but received no answer.

However it is not always easy to know when a question was answered (this was raised several times in the forum):
* for instance for questions whose answers are the default values: was it really answered or not (I would say yes)
* for instance for a group of checkboxes question when no checkbox was checked (difficult to know if this was the choosen response or not)
* for instance for questions having several fields (such as a radio-buttons-group plus a comment-textarea): was it answered if the radiobox is checked but no comment was given ...

Sorry to polute your ticket with my thoughts, but I prefer to write down them immediately as they come so that I do not forget any idea.



TMSWhite (reporter)


We might also want to rename Question Title. From an SPSS and R export perspective, Question Title is really the unique Variable Name, so we should at least make that clear in the documentation or on-line help.

I'm working on (3) soon (just finished (2) and have to do (1) before can do (3)). Hopefully in the next few days.

For #5, we can absolutely add other attributes. I just need a sense of what people would want.

I expect we'd want at least (and probably many more):
(1) isNA(varname) -- returns true if the question was not applicable
(2) isAnswered(varname) --
(3) hasValue(varname) -- this would let us conditionally show TOKEN-related messages only if there is a value for the tokens - e.g. only show the message "{TOKEN:FIRSTNAME}, you have until {EXPIRY} to complete this survey" if both {TOKEN:FIRSTNAME} and {EXPIRY} have values.

I agree that the logic behind isAnswered() can be confusing. We may need a second parameter to let people decide how they want to handle cases where the default value is kept or no selection is made - in such cases you may not know whether the person skipped over that question, or whether they intentionally accept those default values.

One syntax thing to think about is whether we want to use dot notation for attributes, or access them via functions. Although I started with dot notation, there is now more overhead. I'll start with a mix of dot notation and functions and we can always switch this around later if needed.



TMSWhite (reporter)


Now those functions from post 15780 are done in stand-alone mode (but in JavaScript) and with Unit tests. Starting to work on integration so they will control question visibility.



TMSWhite (reporter)

This now works in revision 10579, using much-improved syntax highlighting and tool-tipping


TMSWhite (reporter)

admin pages now show the syntax highlighting for Welcome, Description, Group, Questions, Help, and Relevance (revision 10651).

This should let users immediately see if there are syntax errors, or if they are trying to access a variable that has not been defined.


abita1 (reporter)

A thought on: 15786, #5, ##1,2,3...

Applicable / Not Applicable
Not Answered / Answered
DefaultAnswer / UserSelection
HasValue / LacksValue

Wouldn't these describe an answer space with 2 or 3 dimensions.

Or is there a "default" that is different from a user selecting an answer which is the default-value?

-Issue History
Date Modified Username Field Change
2011-06-16 17:59 TMSWhite New Issue
2011-06-17 12:04 TMSWhite Note Added: 15475
2011-06-18 00:49 TMSWhite Note Added: 15485
2011-06-18 08:53 lemeur Note Added: 15488
2011-06-19 19:03 TMSWhite Relationship added related to 05103
2011-06-19 19:04 TMSWhite Relationship added related to 05268
2011-07-09 20:45 lemeur Note Added: 15706
2011-07-15 21:36 TMSWhite Note Added: 15780
2011-07-15 22:20 lemeur Note Added: 15781
2011-07-16 02:13 TMSWhite Note Added: 15782
2011-07-16 09:02 lemeur Note Added: 15783
2011-07-16 13:35 TMSWhite Note Added: 15784
2011-07-16 13:54 lemeur Note Added: 15785
2011-07-16 18:41 TMSWhite Note Added: 15786
2011-07-19 00:24 TMSWhite Note Added: 15791
2011-07-24 07:26 TMSWhite Note Added: 15846
2011-07-24 17:34 lemeur Note Edited: 15706 View Revisions
2011-08-06 06:22 TMSWhite Note Added: 15989
2012-06-21 12:57 c_schmitz Assigned To => c_schmitz
2012-06-21 12:57 c_schmitz Status new => acknowledged
2012-06-21 12:57 c_schmitz Assigned To c_schmitz =>
2012-06-21 13:22 c_schmitz Project User patches => Development
2012-06-21 13:22 c_schmitz Category Survey design => Survey Design
2012-08-02 07:25 abita1 Note Added: 20111
2012-08-14 22:57 c_schmitz Relationship added has duplicate 05503
+Issue History