the respondent see & e a c u t e ; instead of é
(I have to put spaces between the letter, otherwise, you can't see the "word" that is really displayed)

c_schmitz - the value in the database has the "wrong" value - it contains "Pourquoi {if((boisson.code=='Y'),'aimez-vous le thé','non')} ?"

EM uses htmlspecialchars_decode() when processing database values. Should it be using html_entities_decode(), or should the database be storing non-entity-encoded values?

The database should always store the pure UTF-8 encoded non-entity-encoded value.

OK, so EM is working properly, and somehow the wrong value is being inserted into the database.

Maybe something with database.

I try: importing lss file, and look and save question text.

• Using : no HTML editor, corrected éto é : Work fine.
• Activate inline HTML editor : work fine : in db have é, not é
• Using popup HTML editor, open the HTML editor, don't corrected ( lot of'), save : OK in db.

Then i test with:
Import lss file, direct testing : Not OK.
in db:
aimez-vous le th& eacute ;?
Pourquoi {if((boisson.code=='Y'),'aimez-vous le th& eacute ;','non')} ?
BUT !
LEM change & to & amp;

Did LEM have to accept utf8 encoding AND html encoding ? Then don't replace & to & amp;

Javascript is good:

LEMif(LEManyNA('boisson.code'),'',(LEMif((LEMval('boisson.code') == 'Y'), 'aimez-vous le th& eacute;', 'non')))); (without & amp;)

It's a javascript issue , see some example :
http://www.naveen.com.au/javascript/jquery/encode-or-decode-html-entities-with-jquery/289

TMSWhite: I am sorry, but when I meant 'should' it is not meant in an absolute way. As long as it is valid HTML also entity-encoded is allowed, preferred is non-encoded though.

One core question is how to properly protect EM from cross-site scripting attacks.

I used htmlspecialchars, which takes care of '>','<','"',"'", and '&'. Sounds like we may not need to escape '&'.

However the main issue is that the database is getting the wrong value stored. If you fix the database contents, you'll see the following generated:

LEMif(LEManyNA('boisson.code'),'',(LEMif((LEMval('boisson.code') == 'Y'), 'aimez-vous le thé', 'non'))));

Carsten-

We could have EM process all content through html_entities_decode(), but that is potentially risky. It will work if the original is encoded through html_entities_encode(), but if the original is only processed through html_specialchars(), we might get the wrong result.

It would be nice to ensure we're consistent in the database and always encode/decode with the same function.

I don't understand why we might get the wrong result?

AFAIK htmlentities_decode does not care if encoding with html_specialchars or html_entities_encode was done. If we assume that always valid HTML is supposed to be used it should be fine.

Carsten: it's javascript functionality.

If you make :
< div id="myelement" >< /div >
< script >
document.getElementById("myelement").innerHTML="&";
< /script >

You get:
< div id="myelement" >& amp ;< /div >
and not
< div id="myelement" >&< /div >

EDIT : mlaybe try with https://developer.mozilla.org/fr/JavaScript/R%C3%A9f%C3%A9rence_JavaScript/R%C3%A9f%C3%A9rence_JavaScript/Fonctions_globales/decodeURIComponent

Denis - if so, where in the source code is the problem? Somewhere before insertion into the database, I presume.

Carsten-

You are right - I thought there might be a way to do injection attacks by first encoding with html_spacialchars() and then decoding with html_entities_decode(), but I can't create a working test case of that.

Carsten-

I just tried a dozen permutations of changing some or all of the htmlspecialchars_decode() within EM to html_entity_decode(). All made it worse.

I also remember spending nearly 20 hours last year trying to get the right balance of specialchars and entity encode and decode.

I think we should insist that the database be htmlspecialchars() encoded and not html_entities() encoded. Trying to mix and match the two types of encoding is a big mess.

So, the core fix would be to ensure that entity-encoded data doesn't get into the database.

I try to have html_entities in db, the onky way was to import the lss file and don't change the question text (don't save).

To fix, the only way seem, in em_javascript.js:

/* Display number with comma as radix separator, if needed

/

function LEMfixnum(value)

{

var newval = String(value);

if (parseFloat(newval) != value) {

--        return htmlspecialchars(value);   // unchanged

++        return value;   // really unchanged

}

if (LEMradix===',') {

newval = newval.split('.').join(',');

return newval;

}

return value;

}

I think we need a group decision on this since it affects how we handle cross-site scripting protection.

This we want to automatically do:
(1) Ensure that if a person enters a script into a text entry box, they can't use EM to substitute that script elsewhere (so anything entered in a text entry box should be processed through htmlspecialchars()

Things I'm not sure about:
(1) If people enter markup (like a script) into an equation variable, should it be possible to insert that markup into something else via EM? Perhaps, since the XSS protections tend to protect against accidental creation of markup or scripts via the editor
(2) Similarly,should people be able to write {if(true,'
','')} and have EM insert markup via normal substitution?
(3) If someone enters an answer that contains markup, should the database store the markup as entered, or should it first be processed through htmlspecialchars()?

1) [...]be processed through htmlspecialchars() on output, yes

1.) Think so, yes. It should obey the XSS global filter setting in the admin, though. So if someone tries to save an equation like and the according filter setting is activated it should be filtered accordinlgy.

2.) People? Who is that?

3.) Store as entered. Consider raw data always unsafe to be displayed, but safe to store (assuming that that SQL-injection safe storing methods are used).

For 1 & 2, people = survey authors.

Then 2.) Yes, they should be able to do that.

OK. That isn't how EM currently handles this, so I (or someone else) will need to look through all of the existing calls to htmlspecialchars() within EM and make the needed adjustments.

In general, seems like:
(1) static strings withing expressions should be output without processing through htmlspecialchars()
(2) string content outside of expressions (e.g. text of questions and answers) should be output without processing through htmlspecialchars()
(3) variable values from free-text fields (e.g. a subset of .code, .value, and .shown, including "comment" and "other" fields) should be processed through htmlspecialchars().
(4) tool tips should use htmlentitites() without double encoding.

That proposal sounds fine to me.

attached sample survey for testing special characters. It is possible it won't import correctly. It uses the following text repeatedly:

This <question> has "special chars" including '&'; foreign chars like ßüöäÜÖħéèàçù£, and entities like < > é " ' £ é ò ô

However, that text gets converted at many points, especially during editing. So, consistent handling of entities may be more pervasive than just EM.

Hmm, even Mantis does conversion - the section after "and entities like" spells out the actual entities:

Here they are with the ampersand replaced by a tilde so that they don't get converted:

and entities like ~lt; ~gt; ~eacute; ~quot; ~apos; ~pound; ~eacute; ~ograve; ~ocirc;

Can you point out where it is still going 'wrong' beside EM?

The short answer is that the problem appears to be isolated to EM, but looking at the code, I'm worried it may be more pervasive (which might explain why it took so long to get it "right" in EM in the first place).

When I do a regular expressions search of (/html(specialchars|_entity|entities)/ of 1.92 codebase (searching all .php and .js source, I get 542 matches across 78 files.

Among them, we have several versions of these JavaScript functions:
(1) em_javascript.js and admin_core.js and browse.js have incompatible versions of htmlspecialchars()
(2) em_javascript.js and translation.js have incompatible versions of html_entity_decode()

We should probably move the phpjs.org functions out of em_javascript.js and maintain them separately. At present, I didn't want to touch this since I don't know whether standardizing those functions would break anything.

Also, I'm not clear on some basics, such as, "how should strings be natively stored within":
(1) database
(2) $_SESSION[SGQA]
(3) $fieldmap
(4) other?

And what is the proper way map from one to the other?
(1) database<=>$_SESSION for values
(2) database<=>question "object" for question text, help, relevance, etc. (e.g. database=>$fieldmap)
(3) question object=>HTML nodes
(4) question object=>HTML attributes
(5) $_SESSION values=>HTML input elements
etc.

Furthermore, if we do encoding, should we be doing double encoding and double decoding, or not. For example, "&" => "&amp;"?

My recommendation would be to have someone create succinct coding guidance documentation to indicate the proper way to manage special characters and entities within LS. That way we can (perhaps slowly) look through those 542 matches and make sure they are all correct and consistent.

As mentioned earlier:

Strings should always be stored raw, be it in database, fieldmap or session.

On display:
Strings entered by survey participants should always be htmlspecialchars-encoded (as they are considere unsafe)
Strings entered by survey admin in the administration should not be encoded and displayed as they are (as they are considered to be safe or XSS-filtered)

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=8200

Fix committed to Yii branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=8201

New 1.92+ build released