View Issue Details

This bug affects 1 person(s).
 6
IDProjectCategoryView StatusLast Update
03929Bug reportsOtherpublic2009-12-11 12:17
Reporterpavel Assigned Tomdekker  
PrioritynormalSeveritypartial_block 
Status closedResolutionfixed 
Product Version1.87RC2 
Fixed in Version1.87RC5 
Summary03929: Import data
Description

When I import data from survey and there are several variables from one question
the varibles get names like a7[1],a7[2](a7-the question name)
If I import the data to statistic programs (SAS, SPSS...) it makes me problem with varible names. I think it will be much better to give names like a7_1, a7_2...
Other options in all questions(Hebrew) do more troubles. It's looks like a9[???]. I think the name like a9_other will be much better.
Please see atachment

TagsNo tags attached.
Attached Files
data_import.bmp (1,377,270 bytes)
data_import1.bmp (1,377,270 bytes)
SPSS1.bmp (1,249,398 bytes)
SPSS2.bmp (1,378,134 bytes)
survey_79556_SPSS_data_file.dat (789 bytes)   
'1',,'2','he',,,,,,,,,,,,,,,,,,,,
'2','01-01-1980 00:00:00','1','he','3','3','3','3','1',,'2009-11-04','2','ttttt','3','3','2','2',,'1','hhhh','77','2','1','3'
'3','01-01-1980 00:00:00','1','he','2','1','3','5',,'שדךלגכחדכחדגל','2001-10-12','2','אקספלורר 7','2','1','1',,'המון','2',,'8','3','1',
'4','01-01-1980 00:00:00','1','he','3','1','1','1','1',,'2002-10-24','2','EXPLORER','2','1','1','3',,'2','CVMKJV/','5','2','1','3'
'5','01-01-1980 00:00:00','2','he','1','3','4','4','4',,'2009-03-01','1',,'3','2','2','1',,'1',,'10','1','3','2'
'6','01-01-1980 00:00:00','2','he','1','5','2','1','1',,'2009-11-11','1',,'2','1','1','3',,'2',,'1','1','3','2'
'7','01-01-1980 00:00:00','2','he','3','4','2','2','1',,'2009-11-13','1',,'3','1','1','1',,'1',,'1','1','3','2'
output_spss.doc (1,245 bytes)
my_output_spss.rtf (18,213 bytes)
my_output.sav (5,551 bytes)
Bug heat6
Complete LimeSurvey version number (& build)
I will donate to the project if issue is resolved
Browser
Database type & version
Server OS (if known)
Webserver software & version (if known)
PHP Version

Users monitoring this issue

There are no users monitoring this issue.

Activities

pavel

pavel

2009-11-30 07:52

reporter   ~10333

the same problem in the ranking items variables ( 3 right columns )
It would be a12_1, a12_2, a12_3

user372

2009-11-30 08:11

  ~10334

Last edited: 2009-11-30 08:12

1) Which export type/format and options do you use for export? ==> Please add an export(ed) file for us too!
2) Sorry, I didn't get at a first quick look why and where that's a problem ==> I only use german or english.
3) Please explain once more, perhaps add a screenshot, where we can see where and why there is a problem and add an export file.

pavel

pavel

2009-11-30 08:34

reporter   ~10335

I use excel, becouse other formats create gibberish in Hebrew

pavel

pavel

2009-11-30 08:57

reporter   ~10336

Please see data_import1.bmp.
SAS doesn't permitt signs like "[]", "-" or hebrew letters in the variable names.
When I import data and choose option "question code" and I have more then one variable from a question it creates variables names with the problem signs.
Also you can see hebrew letters in the variabe names from "other options".

user372

2009-11-30 09:28

  ~10337

1) What is SAS? Do you have a link?
2) Is this problem only related to SAS?

pavel

pavel

2009-11-30 09:42

reporter   ~10338

Sorry -))
SAS - is a package for statistical analysis.
I have this problem also in SPSS (other package).
Question codes turn into variable names, but most statistical packages have
restriction for permited signs in the variable names

pavel

pavel

2009-11-30 10:25

reporter   ~10339

Please see "SAS Data Set (Version 8 or 9)" in
http://www.cpc.unc.edu/services/computer/presentations/sasclass99/datafiles.html

user372

2009-11-30 10:31

  ~10340

@ mdekker: as our expert in that topic what do you think about it?

mdekker

mdekker

2009-11-30 11:03

reporter   ~10341

In SPSS export this should not be a problem as variable name validation is done and the name is constructed like proposed. Warnings are put in the code where conflicts are found and the name is adjusted. The basic rule is to choose your question codes and answer values wisely.

If possible, please create an spss export and work from there. If this is not possible, a free alternative would be to use R export as it is basically the same. From there it might be possible to transfer the dataset to SAS, but I have no experience with that.

The excel export used is indeed difficult to use, but I think for excel it is the only way to get meaningful headers as there is only a single dimension to each cell to store questioncode, question, answer while in spss a variable has a name and a label.

The spss output I get form this survey seems ok to me:

RENAME VARIABLE ( V5 = a1_aa1 ).
RENAME VARIABLE ( V6 = a1_aa2 ).
RENAME VARIABLE ( V7 = a1_aa3 ).
RENAME VARIABLE ( V8 = a2 ).
RENAME VARIABLE ( V9 = a3 ).
RENAME VARIABLE ( V10 = a3_other ).
RENAME VARIABLE ( V11 = a4 ).
RENAME VARIABLE ( V12 = a5 ).
RENAME VARIABLE ( V13 = a6 ).
RENAME VARIABLE ( V14 = a7_1 ).
RENAME VARIABLE ( V15 = a7_2 ).
RENAME VARIABLE ( V16 = a8 ).
RENAME VARIABLE ( V17 = a9 ).
RENAME VARIABLE ( V18 = a9_other ).
RENAME VARIABLE ( V19 = a10 ).
RENAME VARIABLE ( V20 = a10_comment ).
RENAME VARIABLE ( V21 = a11 ).
RENAME VARIABLE ( V22 = a12_1 ).
RENAME VARIABLE ( V23 = a12_2 ).
RENAME VARIABLE ( V24 = a12_3 ).

pavel

pavel

2009-11-30 12:12

reporter   ~10343

mdekker, I creates questionnares in Hebrew. When I use SPSS import, all hebrew letters turn into gibberish. Only excel import saves the hebrew data.
Please see atachment

mdekker

mdekker

2009-11-30 12:21

reporter   ~10344

I don't have spss 16 at the moment, stuck with 15 that doesn't handle UTF. When I open the syntax with a text editor I see what looks like hebrew to me. Can you please tell me what version of SPSS you use?

pavel

pavel

2009-11-30 12:25

reporter   ~10345

15

mdekker

mdekker

2009-11-30 13:22

reporter   ~10349

SPSS 15 is not capable of using unicode. Maybe someone with 16 or up can confirm wether the export is correct or not?

To alter the excel export is not up to me I think, maybe ElMatador69 or CSchmitz can assign to the appropriate person?

pavel

pavel

2009-11-30 13:43

reporter   ~10350

when I use SPSS 17 - the same problem

mdekker

mdekker

2009-11-30 13:56

reporter   ~10351

Can you please attach the .sps syntax file for me together with the csv? Please note you can mark the topic as private to ensure your data is safe.

pavel

pavel

2009-11-30 13:59

reporter   ~10352

sure

pavel

pavel

2009-11-30 14:03

reporter   ~10353

I uploaded syntax+*.dat file

mdekker

mdekker

2009-11-30 14:06

reporter   ~10354

As soon as I can get my hands on SPSS 17 I will have a look and see what I can do. Do you get any errors after running the syntax? Sometimes they are 'hidden' in the output so you have to browse the output.

mdekker

mdekker

2009-11-30 14:15

reporter   ~10355

Just a quick thought... The first line reads:
*$Rev: 7762 $.\nSET UNICODE=ON.

Could you delete the first chars so it reads:
SET UNICODE=ON.

and then rerun the syntax?

pavel

pavel

2009-11-30 14:26

reporter   ~10357

In versio 17 I don't see any error, but it runs only the first part of the code

pavel

pavel

2009-11-30 14:27

reporter   ~10358

I uploaded the output

pavel

pavel

2009-11-30 14:33

reporter   ~10359

I tried to run only SET UNICODE=ON - the same result

mdekker

mdekker

2009-11-30 14:34

reporter   ~10361

Last edited: 2009-11-30 14:35

I attached my output from spss15 and my sav file. Because it doesn't have unicode support i have an error for that line and i suppose the encoding is crap but i get the correct variable names.

You should put set unicode first and then run the rest of the script (run the complete script including set unicode as the first line)

mdekker

mdekker

2009-11-30 14:37

reporter   ~10362

Attached the sps file that should run ok (maybe you need to set the path to the dat file first)

pavel

pavel

2009-11-30 14:45

reporter   ~10364

I tried your syntax - it runs until VARIABLE LABELS, as before

pavel

pavel

2009-11-30 14:50

reporter   ~10365

Ok, I the data same to your ,but still instead of Hebrew I see other symbols

pavel

pavel

2009-11-30 14:51

reporter   ~10366

got

pavel

pavel

2009-11-30 14:54

reporter   ~10368

Sorry, I ran it again and now it's ok!!!!

Thank you -)

pavel

pavel

2009-11-30 14:56

reporter   ~10369

What did you changed in the syntax?Only fist row?

mdekker

mdekker

2009-11-30 14:57

reporter   ~10370

I so I guess the first line was the one that gave the problems. I though I fixed it already but that doesn't seem the case. I will fix it in svn today.

mdekker

mdekker

2009-11-30 15:03

reporter   ~10371

The only change was the first line. I just committed the fix to svn if you are familiar with that. If not, you can edit the \admin\export_data_spss.php to read:

$headerComment = '*$Rev: 8020 $.' . "\n";

on line 34.

mdekker

mdekker

2009-11-30 15:04

reporter   ~10372

Fixed in revision 8020, incorrect newline prevented the unicode line from running.

Issue History

Date Modified Username Field Change
2009-11-30 07:48 pavel New Issue
2009-11-30 07:48 pavel Status new => assigned
2009-11-30 07:48 pavel Assigned To => user372
2009-11-30 07:48 pavel File Added: data_import.bmp
2009-11-30 07:52 pavel Note Added: 10333
2009-11-30 08:11 user372 Note Added: 10334
2009-11-30 08:12 user372 Note Edited: 10334
2009-11-30 08:12 user372 Status assigned => feedback
2009-11-30 08:34 pavel File Added: results-survey79556(1).xls
2009-11-30 08:34 pavel Note Added: 10335
2009-11-30 08:41 pavel File Added: limesurvey_survey_79556.csv
2009-11-30 08:41 pavel File Added: data_import1.bmp
2009-11-30 08:57 pavel Note Added: 10336
2009-11-30 09:28 user372 Note Added: 10337
2009-11-30 09:42 pavel Note Added: 10338
2009-11-30 10:25 pavel Note Added: 10339
2009-11-30 10:31 user372 Status feedback => assigned
2009-11-30 10:31 user372 Assigned To user372 => mdekker
2009-11-30 10:31 user372 Note Added: 10340
2009-11-30 11:03 mdekker Note Added: 10341
2009-11-30 12:12 pavel Note Added: 10343
2009-11-30 12:12 pavel File Added: SPSS1.bmp
2009-11-30 12:15 pavel File Added: SPSS2.bmp
2009-11-30 12:21 mdekker Note Added: 10344
2009-11-30 12:25 pavel Note Added: 10345
2009-11-30 13:22 mdekker Note Added: 10349
2009-11-30 13:43 pavel Note Added: 10350
2009-11-30 13:56 mdekker Note Added: 10351
2009-11-30 13:59 pavel Note Added: 10352
2009-11-30 14:02 pavel File Added: survey_79556_SPSS_data_file.dat
2009-11-30 14:03 pavel Note Added: 10353
2009-11-30 14:04 pavel File Added: survey_79556_SPSS_syntax_file.sps
2009-11-30 14:06 mdekker Note Added: 10354
2009-11-30 14:15 mdekker Note Added: 10355
2009-11-30 14:25 pavel File Added: output_spss.doc
2009-11-30 14:26 pavel Note Added: 10357
2009-11-30 14:27 pavel Note Added: 10358
2009-11-30 14:32 mdekker File Added: my_output_spss.rtf
2009-11-30 14:33 mdekker File Added: my_output.sav
2009-11-30 14:33 pavel Note Added: 10359
2009-11-30 14:34 mdekker Note Added: 10361
2009-11-30 14:34 mdekker Note Edited: 10361
2009-11-30 14:35 mdekker Note Edited: 10361
2009-11-30 14:37 mdekker File Added: my_survey_79556_SPSS_syntax_file.sps
2009-11-30 14:37 mdekker Note Added: 10362
2009-11-30 14:45 pavel Note Added: 10364
2009-11-30 14:50 pavel Note Added: 10365
2009-11-30 14:51 pavel Note Added: 10366
2009-11-30 14:54 pavel Note Added: 10368
2009-11-30 14:56 pavel Note Added: 10369
2009-11-30 14:57 mdekker Note Added: 10370
2009-11-30 15:03 mdekker Note Added: 10371
2009-11-30 15:04 mdekker Note Added: 10372
2009-11-30 15:04 mdekker Status assigned => resolved
2009-11-30 15:04 mdekker Fixed in Version => 1.87RC5
2009-11-30 15:04 mdekker Resolution open => fixed
2009-12-11 12:17 c_schmitz Status resolved => closed
2010-05-06 10:27 c_schmitz Category Import / Export => (No Category)