View Issue Details

This bug affects 1 person(s).
 4
IDProjectCategoryView StatusLast Update
03471Bug reportsOtherpublic2009-07-29 11:11
Reportershin2 Assigned Tomdekker  
PrioritynormalSeverityminor 
Status closedResolutionfixed 
Product Version1.85+ 
Fixed in Version1.85+ 
Summary03471: Data from radio lists with alphanumeric labels is incorrectly exported to R and SPSS
Description

Given that a list (radio) has 3 options with alphanumeric answer codes a1, a2, and a3, the R and SPSS data export is incorrect.

R: the csv file with the data encodes the answers numerically, e.g., 1, 2, or 3. The numerical vector that contains the data after read.table() is later converted to a factor with levels equal to the LimeSurvey answer codes (a1, a2, a3) -> data$V4=factor(data$V4,levels=c("a1","a2","a3"),labels=c("Answer1","Answer2","Answer3")). Since the numerical values are not among the given levels, they are changed to NA (missing value).

SPSS: the dat file with the data records the answer codes - alphanumeric in the given case. The syntax file that describes the data matrix defines the corresponding variable as having F2 format (numerical with width 2). As a result, all values are recorded as missing. In the given case, F2 would need to be replaced by A2.

TagsNo tags attached.
Attached Files
limesurvey_survey_export.csv (3,662 bytes)   
# LimeSurvey Survey Dump
# DBVersion 138
# This is a dumped survey from the LimeSurvey Script
# http://www.limesurvey.org/
# Do not change this header!

#
# SURVEYS TABLE
#
"sid","owner_id","admin","active","expires","startdate","adminemail","private","faxto","format","template","language","additional_languages","datestamp","usecookie","notification","allowregister","allowsave","autonumber_start","autoredirect","allowprev","printanswers","ipaddr","refurl","datecreated","publicstatistics","publicgraphs","listpublic","htmlemail","tokenanswerspersistence","assessments","usecaptcha","usetokens","bounce_email","attributedescriptions"
"66249","1","Your Name","Y","","","your@email.org","Y","","G","default","en","","N","N","0","N","Y","0","N","Y","N","N","N","2009-07-11","N","N","N","Y","N","N","D","N","your@email.org",""

#
# GROUPS TABLE
#
"gid","sid","group_name","group_order","description","language"
"29","66249","G01","0","","en"

#
# QUESTIONS TABLE
#
"qid","sid","gid","type","title","question","preg","help","other","mandatory","lid","lid1","question_order","language"
"210","66249","29","L","Q01_01","Question<br />","","","N","N","0","0","0","en"

#
# ANSWERS TABLE
#
"qid","code","answer","default_value","assessment_value","sortorder","language"
"210","a1","Answer1","N","0","1","en"
"210","a2","Answer2","N","0","2","en"
"210","a3","Answer3","N","0","3","en"

#
# CONDITIONS TABLE
#

#
# LABELSETS TABLE
#

#
# LABELS TABLE
#

#
# QUESTION_ATTRIBUTES TABLE
#

#
# ASSESSMENTS TABLE
#

#
# SURVEYS_LANGUAGESETTINGS TABLE
#
"surveyls_survey_id","surveyls_language","surveyls_title","surveyls_description","surveyls_welcometext","surveyls_endtext","surveyls_url","surveyls_urldescription","surveyls_email_invite_subj","surveyls_email_invite","surveyls_email_remind_subj","surveyls_email_remind","surveyls_email_register_subj","surveyls_email_register","surveyls_email_confirm_subj","surveyls_email_confirm","surveyls_dateformat"
"66249","en","Survey","","","<br />","","","Invitation to participate in survey","Dear {FIRSTNAME},\n\nYou have been invited to participate in a survey.\n\nThe survey is titled:\n""{SURVEYNAME}""\n\n""{SURVEYDESCRIPTION}""\n\nTo participate, please click on the link below.\n\nSincerely,\n\n{ADMINNAME} ({ADMINEMAIL})\n\n----------------------------------------------\nClick here to do the survey:\n{SURVEYURL}","Reminder to participate in survey","Dear {FIRSTNAME},\n\nRecently we invited you to participate in a survey.\n\nWe note that you have not yet completed the survey, and wish to remind you that the survey is still available should you wish to take part.\n\nThe survey is titled:\n""{SURVEYNAME}""\n\n""{SURVEYDESCRIPTION}""\n\nTo participate, please click on the link below.\n\nSincerely,\n\n{ADMINNAME} ({ADMINEMAIL})\n\n----------------------------------------------\nClick here to do the survey:\n{SURVEYURL}","Survey registration confirmation","Dear {FIRSTNAME},\n\nYou, or someone using your email address, have registered to participate in an online survey titled {SURVEYNAME}.\n\nTo complete this survey, click on the following URL:\n\n{SURVEYURL}\n\nIf you have any questions about this survey, or if you did not register to participate and believe this email is in error, please contact {ADMINNAME} at {ADMINEMAIL}.","Confirmation of completed survey","Dear {FIRSTNAME},\n\nThis email is to confirm that you have completed the survey titled {SURVEYNAME} and your response has been saved. Thank you for participating.\n\nIf you have any further questions about this email, please contact {ADMINNAME} at {ADMINEMAIL}.\n\nSincerely,\n\n{ADMINNAME}","1"

#
# QUOTA TABLE
#

#
# QUOTA_MEMBERS TABLE
#

limesurvey_survey_export.csv (3,662 bytes)   
survey_66249_data_file.csv (36 bytes)   
'1','01-01-1980 00:00:00','en','a1'
Bug heat4
Complete LimeSurvey version number (& build)7253
I will donate to the project if issue is resolved
BrowserFirefox 3.0.11
Database type & versionMySQL 5.0.45
Server OS (if known)Windows XP SP3
Webserver software & version (if known)Apache 2.2.* (XAMPP 1.6.4 Windows)
PHP Version5.2.4

Users monitoring this issue

There are no users monitoring this issue.

Activities

mdekker

mdekker

2009-07-13 09:41

reporter   ~08825

The csv and the dat file are identical to me (as they should be because they are generated with the same function). But there is a problem with the variable type being numerical instead of alpha. This is a bug caused by performance improvements and should be fixed by replacing your export_data_functions.php with the one I attached. Thanks for pointing it out.

Could you please verify this file is working for you?

shin2

shin2

2009-07-13 14:12

reporter   ~08827

Last edited: 2009-07-13 14:13

Thanks for your very quick fix! With the attached file, everything works.

BTW, not a bug, just a suggestion: the R syntax file is named Surveydata_syntax.R, i.e., without a surveySID prefix like the other exported files. If there is no specific reason for doing so, I suggest adding that prefix also for the R syntax file.

Less than a suggestion: the comment.char argument to read.table() in R has a default value of "#". I don't know if # can be exported from LimeSurvey without being enclosed in ' ', but if so, explicitly setting comment.char="" in read.table() might be an option.

Many thanks for your work!

mdekker

mdekker

2009-07-13 16:00

reporter   ~08828

Last edited: 2009-07-13 16:05

I guess your first suggestion could be implemented very easy and makes it easier to keep your different files apart. I will add that when I commit this fix to the repository. But it has impact on the translations so I will have to ask first before I can commit the patch.

For the second one, with the comment.char... Is it something that you need? I don't use R myself, but adding that options doesn't do anything for me. Can you please give me an easy example where adding this option makes a difference?

mdekker

mdekker

2009-07-13 16:10

reporter   ~08829

@c_schmitz: Changing the name of the R export syntax to a default survey_sid prefix involves a problem with the translation. The name is literally in the translation string so when I change the text... all translations will be broken.

Can you advise how to proceed? Should I commit the name change to the dev branch and the fix for the problem to the normal one?

shin2

shin2

2009-07-13 19:11

reporter   ~08834

Re: comment.char
This is not something I need, i.e., no problem I have come across. Comments would only be a problem if there was a way for a # character to end up in the exported data without being enclosed in quotes. In that case, read.table() would not interpret the part of a line coming after #. To prevent that from happening, the comment.char="" argument turns off interpretation of comments - as is the default in R's read.csv() function which is simply read.table() with different default arguments.

BTW, the colClasses argument to read.table() promises performance benefits for large data files - it is used to describe the type of each variable, similar to the SPSS syntax. If there is no colClasses argument, R guesses the type itself which can take some time.

mdekker

mdekker

2009-07-14 09:20

reporter   ~08836

Ah now I understand what you mean :) So the comment.char is just for safety. I found this page: http://www.biostat.jhsph.edu/~rpeng/docs/R-large-tables.html on the net and ask our R expert to have a look at that. We might be able to change the export to accommodate for these changes.

Thanks for your help! Are you interested in helping to test possible changes to the R export? If you send me your email in a private message I can send you the updated files to test. I will open a feature request for this so we can close this bug and continue in the new feature request :)

http://bugs.limesurvey.org/view.php?id=3477

mdekker

mdekker

2009-07-14 13:15

reporter   ~08842

Submitted just the patch to this problem in rev. 7315 and leave the rest to be addressed in the related feature request.

Issue History

Date Modified Username Field Change
2009-07-11 19:15 shin2 New Issue
2009-07-11 19:15 shin2 Status new => assigned
2009-07-11 19:15 shin2 Assigned To => user372
2009-07-11 19:15 shin2 File Added: limesurvey_survey_export.csv
2009-07-11 19:15 shin2 Build Number => 7253
2009-07-11 19:15 shin2 Browser => Firefox 3.0.11
2009-07-11 19:15 shin2 Database & DB-Version => MySQL 5.0.45
2009-07-11 19:15 shin2 Operating System (Server) => Windows XP SP3
2009-07-11 19:15 shin2 Webserver => Apache 2.2.* (XAMPP 1.6.4 Windows)
2009-07-11 19:15 shin2 PHP Version => 5.2.4
2009-07-11 19:22 shin2 File Added: survey_66249_data_file.csv
2009-07-11 19:23 shin2 File Added: survey_66249_SPSS_data_file.dat
2009-07-11 19:23 shin2 File Added: survey_66249_SPSS_syntax_file.sps
2009-07-11 19:23 shin2 File Added: survey_66249_syntax.R
2009-07-11 19:57 user372 Assigned To user372 => mdekker
2009-07-13 09:37 mdekker File Added: export_data_functions.php
2009-07-13 09:41 mdekker Note Added: 08825
2009-07-13 09:41 mdekker Status assigned => feedback
2009-07-13 14:12 shin2 Note Added: 08827
2009-07-13 14:13 shin2 Note Edited: 08827
2009-07-13 16:00 mdekker Note Added: 08828
2009-07-13 16:05 mdekker Note Edited: 08828
2009-07-13 16:06 mdekker Status feedback => assigned
2009-07-13 16:06 mdekker Assigned To mdekker => c_schmitz
2009-07-13 16:10 mdekker Note Added: 08829
2009-07-13 19:11 shin2 Note Added: 08834
2009-07-14 09:20 mdekker Note Added: 08836
2009-07-14 13:14 mdekker Assigned To c_schmitz => mdekker
2009-07-14 13:15 mdekker Note Added: 08842
2009-07-14 13:15 mdekker Status assigned => resolved
2009-07-14 13:15 mdekker Fixed in Version => 1.85+
2009-07-14 13:15 mdekker Resolution open => fixed
2009-07-29 11:11 c_schmitz Status resolved => closed
2010-05-06 10:27 c_schmitz Category Import / Export => (No Category)