View Issue Details

IDProjectCategoryView StatusLast Update
13904Bug reports[All Projects] Survey designpublic2019-05-29 16:39
ReporterefingdmiAssigned Toc_schmitz 
PriorityhighSeveritymajor 
Status closedResolutionfixed 
Product Version3.13.x 
Target VersionFixed in Version3.17.x 
Summary13904: Japanese characters get replaced with '?'
Description

Settings -> Text elements (Welcome message, Description, End, Privacy...)

If I enter a japanese text here, it will be replaced with a lot of questionmarks. This happens only in the editor windows on this page. Everything is fine e.g. when creating questions.
Due to that, I think it may be a bug and not a problem of my settings.

Steps To Reproduce

Settings -> Text elements (Welcome message, Description, End, Privacy...) -> enter japanese characters
survey attached

TagsNo tags attached.
Complete LimeSurvey version number (& build)3.13.2+180709
I will donate to the project if issue is resolvedNo
BrowserFF, Chrome
Database & DB-VersionMS SQL Server 2012
Server OS (if known)Windows Server 2012 R2 64Bit
Webserver software & version (if known)Apache Apache 2.4.33 (64bit)
PHP VersionPHP 7.2.6 (64bit)

Relationships

related to 14545 closedDenisChenu Unable to reset menu entries with MSSQL 
has duplicate 14063 closedc_schmitz Arabic text is corrupted in several sections in admin view and user view 
related to 13989 closedcdorin Russian characters are getting displayed as question marks 

Activities

efingdmi

efingdmi

2018-07-25 07:24

reporter  

before.png (59,883 bytes)
before.png (59,883 bytes)
after.png (70,486 bytes)
after.png (70,486 bytes)
questions.png (15,384 bytes)
questions.png (15,384 bytes)
limesurvey_survey_116537.lss (13,726 bytes)
efingdmi

efingdmi

2018-08-20 09:41

reporter   ~48823

This also occurs if someone answers in Japanese. So the results are not usable :(

jelo

jelo

2018-08-22 14:43

partner   ~48853

Have you used the language before with this installation of LimeSurvey?
Or is this the first time you use Japanese characters in this LimeSurvey installation?

Did I get it right. The Umfragetitel is always shown correctly? Any other fields, which are showing the correct content after saving the form?

Apache and PHP are running on Windows, right?

efingdmi

efingdmi

2018-08-23 08:07

reporter   ~48857

No it was the first time we tried Japanese characters.
The survey title works fine, the questions are also shown in Japanese correctly.
But if I save the fields mentioned above and if someone answers the questions and I check the answers I see a lot of "?"

Maybe some probem with mssql? I had no chance to test it with mysql, sorry :(

jelo

jelo

2018-08-23 09:59

partner   ~48859

Can you check what admin theme is set? Configuration -> Themes -> Admin themes.

There is a admin theme
"Default LimeSurvey admin interface template with NOTO font. NOTO font is will provide a correct display for any existing language."

To me it looks more like an issue which LS only has when interacting with MS SQL.
But would be nice if you could try the NOTO theme with a few browsers and check if you sense a difference.

efingdmi

efingdmi

2018-08-23 10:39

reporter   ~48861

I will try.
Default installation of V 2.62.2 is working on the same database

efingdmi

efingdmi

2018-08-23 10:44

reporter   ~48862

No change when switching to the NOTO default template.

Version 2.62.2+170203 is working perfect in the same database. Even with the SeaGreen template. Maybe this helps.

LouisGac

LouisGac

2018-08-28 11:21

manager   ~48888

markus, I'd like your point of view on this problem

efingdmi

efingdmi

2018-09-10 13:49

reporter   ~48999

anything I could do to help here?

efingdmi

efingdmi

2018-11-30 14:45

reporter   ~49856

still nothing? I searched a lot, compared it with 2.62.2 but I did not find the solution :(

DenisChenu

DenisChenu

2018-12-18 00:33

developer   ~49980

Surely related to DB (MS SQL Server 2016) and maybe Server OS (Windows Server 2012 R2 64Bit)

We already use nvarchar and ntext for MSSQL :https://stackoverflow.com/a/22955415/2239406

About the "how to find when this issue happen"
With a working 2.62 version, update to a https://github.com/LimeSurvey/LimeSurvey/releases/tag/2.72.3%2B171020 and see if you have same issue.
If yes, come back to 2.62 and go to https://github.com/LimeSurvey/LimeSurvey/releases/tag/2.63.0%2B170304 , if OK : https://github.com/LimeSurvey/LimeSurvey/releases/tag/2.63.1%2B170305 etc …

Until find the broken release.

efingdmi

efingdmi

2018-12-18 14:05

reporter   ~49986

Hi, and thanks for the reply.
Currently we are using 3.13 and the issue appears. In the database I see that the data is stored in 'text', not in ntext

Just to clarify: I should set up a secound system again (2.62) and upgrade it step by step? Then I will try it :)

DenisChenu

DenisChenu

2019-01-09 12:04

developer   ~50132

Can not reproduce on 3.15.5 version on Windows server 2016 (IIS) + php 7.2.9 + sqlsrv + msodbcsql17.dll + SQLServer 12.00.5590
With firefox on windows

efingdmi

efingdmi

2019-01-15 10:21

reporter   ~50206

I will try this asap :)

efingdmi

efingdmi

2019-01-21 13:13

reporter   ~50286

stil the same :(



japanese.png (60,928 bytes)
japanese.png (60,928 bytes)
efingdmi

efingdmi

2019-02-12 17:37

reporter   ~50533

It is working in 2.27.3, but not in 3.x

Difference I found in application\core\db\MssqlSchema,php
beginning from line 50 it is different between ls version 2. and 3.
could it be, that there is something wrong? I'm honest, I did not take a deeper look yet

efingdmi

efingdmi

2019-02-12 17:46

reporter   ~50535

Both versions have the line: $this->columnTypes['text'] = 'nvarchar(max)';
But in 3.X the columns are still created as text and not as nvarchar

DenisChenu

DenisChenu

2019-02-13 09:36

developer   ~50537

This part : https://github.com/LimeSurvey/LimeSurvey/blob/master/application/core/db/MssqlSchema.php#L50-L66

Here is the same function used in 2.73 : https://github.com/SondagesPro/LimeSurvey-SondagesPro/blob/2.73/framework/db/schema/CDbSchema.php#L371-L383

It's seems to update ONLY for pk, then in think it's a difference in your MSSQL server

efingdmi

efingdmi

2019-02-13 09:55

reporter   ~50540

hmm true. Can't find a difference.

I'm using the same databse for the two versions, also the same settings.

markusfluer

markusfluer

2019-02-15 13:45

administrator   ~50568

Is the database collation the same?
Typically Limesurvey tries to use UTF8 as much as possible.

DenisChenu

DenisChenu

2019-02-15 16:43

developer   ~50577

@markusfluer : mssql issue somewhere
It's the reason of nvarchar : https://www.mssqltips.com/sqlservertip/4322/sql-server-differences-of-char-nchar-varchar-and-nvarchar-data-types/

And this line : https://github.com/LimeSurvey/LimeSurvey/blob/b96ff7cba0ecff5b80a912f762903bc12c6c4ce0/application/core/db/MssqlSchema.php#L19 or https://github.com/LimeSurvey/LimeSurvey/blob/b96ff7cba0ecff5b80a912f762903bc12c6c4ce0/application/core/db/MssqlSchema.php#L11

efingdmi

efingdmi

2019-02-17 12:05

reporter   ~50588

@markusfluer everything is the same but the LS Version, and the table prefix, lime26 and lime3, so there shouldn't be the problem.

efingdmi

efingdmi

2019-02-18 15:39

reporter   ~50608

it is a MS SQL Server 2012, can't change it in the Bug report

c_schmitz

c_schmitz

2019-05-23 12:28

administrator   ~52070

Will be fixed in next version. Existing databases will be automatically updated to fix the issue.

efingdmi

efingdmi

2019-05-23 15:09

reporter   ~52082

awesome news, thank you so much

c_schmitz

c_schmitz

2019-05-29 16:12

administrator   ~52172

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=28883

c_schmitz

c_schmitz

2019-05-29 16:39

administrator   ~52232

Version 3.17.4+190529 released

Related Changesets

LimeSurvey: master 64ba8d40

2019-05-23 12:01:04

c_schmitz

Details Diff
Fixed issue 13904: UTF-8 characters not correctly saved in survey texts when using MSSQL DB Affected Issues
13904
mod - application/config/version.php Diff File
mod - application/helpers/update/updatedb_helper.php Diff File
mod - installer/create-database.php Diff File

Issue History

Date Modified Username Field Change
2018-07-25 07:24 efingdmi New Issue
2018-07-25 07:24 efingdmi File Added: before.png
2018-07-25 07:24 efingdmi File Added: after.png
2018-07-25 07:24 efingdmi File Added: questions.png
2018-07-25 07:24 efingdmi File Added: limesurvey_survey_116537.lss
2018-08-20 09:41 efingdmi Note Added: 48823
2018-08-22 12:26 LouisGac Priority none => high
2018-08-22 12:26 LouisGac Severity minor => major
2018-08-22 13:29 jelo Relationship added related to 13989
2018-08-22 14:43 jelo Note Added: 48853
2018-08-23 08:07 efingdmi Note Added: 48857
2018-08-23 09:59 jelo Note Added: 48859
2018-08-23 10:39 efingdmi Note Added: 48861
2018-08-23 10:44 efingdmi Note Added: 48862
2018-08-28 11:21 LouisGac Assigned To => markusfluer
2018-08-28 11:21 LouisGac Status new => assigned
2018-08-28 11:21 LouisGac Note Added: 48888
2018-09-10 13:49 efingdmi Note Added: 48999
2018-11-30 14:45 efingdmi Note Added: 49856
2018-12-18 00:33 DenisChenu Note Added: 49980
2018-12-18 14:05 efingdmi Note Added: 49986
2019-01-09 12:04 DenisChenu Note Added: 50132
2019-01-15 10:21 efingdmi Note Added: 50206
2019-01-21 13:13 efingdmi File Added: japanese.png
2019-01-21 13:13 efingdmi Note Added: 50286
2019-02-12 17:37 efingdmi Note Added: 50533
2019-02-12 17:46 efingdmi Note Added: 50535
2019-02-13 09:36 DenisChenu Note Added: 50537
2019-02-13 09:55 efingdmi Note Added: 50540
2019-02-15 13:45 markusfluer Note Added: 50568
2019-02-15 16:43 DenisChenu Note Added: 50577
2019-02-17 12:05 efingdmi Note Added: 50588
2019-02-18 15:39 efingdmi Note Added: 50608
2019-02-18 16:01 DenisChenu Database & DB-Version MS SQL Server 2016 => MS SQL Server 2012
2019-02-19 09:28 DenisChenu Relationship added related to 14545
2019-05-23 12:26 c_schmitz Relationship added has duplicate 14063
2019-05-23 12:26 c_schmitz Assigned To markusfluer => c_schmitz
2019-05-23 12:28 c_schmitz Status assigned => resolved
2019-05-23 12:28 c_schmitz Resolution open => fixed
2019-05-23 12:28 c_schmitz Fixed in Version => 3.17.x
2019-05-23 12:28 c_schmitz Note Added: 52070
2019-05-23 15:09 efingdmi Note Added: 52082
2019-05-29 16:12 c_schmitz Changeset attached => LimeSurvey master 64ba8d40
2019-05-29 16:12 c_schmitz Note Added: 52172
2019-05-29 16:39 c_schmitz Note Added: 52232
2019-05-29 16:39 c_schmitz Status resolved => closed