View Issue Details

This bug affects 1 person(s).
 12
IDProjectCategoryView StatusLast Update
05126Bug reportsImport/Exportpublic2012-03-14 21:08
Reporterngeorgiev Assigned Toc_schmitz  
PrioritynormalSeverityminor 
Status closedResolutionunable to reproduce 
Product Version1.91RC6 
Summary05126: CSV Export - no new lines
Description

I want to parse an exported CSV survey data, but it does not contain the new lines entered by the user.

In my survey many users enter new lines and I need them for parsing. But they are not saved in the CSV file, so I have to add them manually (comparing the data with the notification email which contains the new lines). But it takes me at least 20 minutes for each survey entry!

Can you add an option which allows the new lines to be saved in the exported CSV file?

A recommendation from Mazi was to "add another filter option like "filter linebreaks".
http://www.limesurvey.org/forum/installation-a-update-issues/56512-csv-export-no-new-lines-problem?lang=en#56595

Steps To Reproduce

. Make a survey with a question with long text
. Fill out the survey by entering text and several new lines in the long text field
. Export the survey entries to .CSV and check that the file does not contain the entered new lines.

Additional Information

The bug is described also in the forum:
http://www.limesurvey.org/forum/installation-a-update-issues/56512-csv-export-no-new-lines-problem

TagsNo tags attached.
Attached Files
csv.fix.patch (4,464 bytes)   
Index: admin/export_data_functions.php
===================================================================
--- admin/export_data_functions.php	(revision 9992)
+++ admin/export_data_functions.php	(working copy)
@@ -29,6 +29,21 @@
 }
 
 /**
+ * Strips html tags
+ *
+ * @param $string
+ * @return $string
+ */
+function strip_tags_full_save_newline($string) {
+    $string=html_entity_decode($string, ENT_QUOTES, "UTF-8");
+    //combining these into one mb_ereg_replace call ought to speed things up
+    //$string = str_replace(array("\r\n","\r","\n",'-oth-'), '', $string);
+    //The backslashes must be escaped twice, once for php, and again for the regexp
+    //$string = str_replace("'|\\\\'", "'", $string);
+    return FlattenTextWithNewline($string);
+}
+
+/**
  * Returns true if passed $value is numeric
  *
  * @param $value
@@ -46,9 +61,8 @@
     return ($eng_or_world);
 }
 
-function spss_export_data ($na = null) {
+function spss_export_data ($na = null, $is_csv = false) {
     global $length_data;
-
     // Build array that has to be returned
     $fields = spss_fieldmap();
 
@@ -138,9 +152,19 @@
                     echo("'0'");
                 }
             } elseif (!$field['hide']) {
-                $strTmp=mb_substr(strip_tags_full($row[$fieldno]), 0, $length_data);
+				if($is_csv == true) {
+					$strTmp=mb_substr(strip_tags_full_save_newline($row[$fieldno]), 0, $length_data);
+				}
+				else {
+					$strTmp=mb_substr(strip_tags_full($row[$fieldno]), 0, $length_data);
+				}
                 if (trim($strTmp) != ''){
-                    $strTemp=str_replace(array("'","\n","\r"),array("''",' ',' '),trim($strTmp));
+					if($is_csv == true) {
+						$strTemp=str_replace(array("'"),array("''"),trim($strTmp));
+					}
+					else {
+						$strTemp=str_replace(array("'","\n","\r"),array("''",' ',' '),trim($strTmp));
+					}
                     /*
                      * Temp quick fix for replacing decimal dots with comma's
                      if (my_is_numeric($strTemp)) {
Index: admin/export_data_r.php
===================================================================
--- admin/export_data_r.php	(revision 9992)
+++ admin/export_data_r.php	(working copy)
@@ -138,7 +138,7 @@
     header("Pragma: public");
 
     $na="";	//change to empty string instead of two double quotes to fix warnings on NA
-    spss_export_data($na);
+    spss_export_data($na, true); // this true indicates that the format is .csv
 
     exit;
 }
Index: common_functions.php
===================================================================
--- common_functions.php	(revision 9992)
+++ common_functions.php	(working copy)
@@ -5008,7 +5008,7 @@
     $sNicetext = strip_javascript($sTextToFlatten);
     $sNicetext = strip_tags($sNicetext);
     $sNicetext = str_replace(array("\n","\r"),array('',''), $sNicetext);
-    if ($bDecodeHTMLEntities==true)
+	if ($bDecodeHTMLEntities==true)
     {
         $sNicetext = str_replace(' ',' ', $sNicetext); // html_entity_decode does not properly convert   to spaces
         $sNicetext = html_entity_decode($sNicetext, ENT_QUOTES, $sCharset);
@@ -5019,6 +5019,30 @@
 
 
 /**
+ *  This functions removes all HTML tags, Javascript, and other strange chars from a given text
+ *
+ * @param string $sTextToFlatten  Text you want to clean
+ * @param boolan $bDecodeHTMLEntities If set to true then all HTML entities will be decoded to the specified charset. Default: false
+ * @param string $sCharset Charset to decode to if $decodeHTMLEntities is set to true
+ * 
+ * @return string  Cleaned text
+ */
+function FlattenTextWithNewline($sTextToFlatten, $bDecodeHTMLEntities=false, $sCharset='UTF-8')
+{
+    $sNicetext = strip_javascript($sTextToFlatten);
+    $sNicetext = strip_tags($sNicetext);
+    $sNicetext = str_replace(array("\r\n","\r","\n"),array(PHP_EOL,PHP_EOL,PHP_EOL), $sNicetext);
+	if ($bDecodeHTMLEntities==true)
+    {
+        $sNicetext = str_replace(' ',' ', $sNicetext); // html_entity_decode does not properly convert   to spaces
+        $sNicetext = html_entity_decode($sNicetext, ENT_QUOTES, $sCharset);
+    }
+    $sNicetext = trim($sNicetext);
+    return  $sNicetext;
+}
+
+
+/**
  * getArrayFiltersForGroup() queries the database and produces a list of array_filter questions and targets with in the same group
  * @global string $surveyid
  * @global string $gid
csv.fix.patch (4,464 bytes)   
Bug heat12
Complete LimeSurvey version number (& build)9642
I will donate to the project if issue is resolvedYes
Browser
Database type & versionmysql
Server OS (if known)Linux
Webserver software & version (if known)unknown
PHP Versionshould be more than 5

Users monitoring this issue

mot

Activities

pnarula

pnarula

2011-04-27 19:19

reporter   ~14867

Carsten, please have a look.

pnarula

pnarula

2011-04-28 11:23

reporter   ~14869

Reminder sent to: c_schmitz

Please review this patch.

Also do I need to assign the issue back to you or a reminder would be sufficient?

mot

mot

2011-06-03 11:36

reporter   ~15270

I could not understand Mazi's comment about "filter linebreaks", as in

I read it this way: Upon export to CSV, and option is given to filter linebreaks, like a checkbox [X] Filter Linebreaks

If the option is ON, linebreaks will be filtered (removed/converted to space).

If the option is OFF, linebreaks will be put into the CSV.

So if some CSV readers don't support linebreaks when reading in CSV data, will not get broken.

Ref: http://en.wikipedia.org/wiki/Comma-separated_values

in specific:

  • fields that contain commas, double-quotes, or line-breaks must be quoted.
  • a line break within an element must be preserved.

So from a standards point of view, line breaks should be supported. But not by replacing them with {break} but just by properly writing the CSV output.

Maybe what you suggest in the patch is a third way of doing so.

For the moment I would say:

1.) The current implementation is broken as it does not follow the CSV specs.
2.) The implementation can be fixed, and an option can be given to mimic the old behavior.
3.) A third implementation could be written, that replaces control characters with placeholders. However this is only of very limited use.

c_schmitz

c_schmitz

2011-06-05 21:15

administrator   ~15303

Pooja, I concur with mot's opinion. Can you create an according patch vs. stable please?

c_schmitz

c_schmitz

2011-06-16 14:39

administrator   ~15465

?

pnarula

pnarula

2011-06-16 23:31

reporter   ~15470

Uploaded the patch following the csv specifications.

c_schmitz

c_schmitz

2011-06-19 15:09

administrator   ~15507

pnarula, the patch is vs. unstable. Please create a patch vs the stable branch.

c_schmitz

c_schmitz

2011-06-20 16:36

administrator   ~15526

Thank you pnarula,

can you change the patch by extending the existing functions like FlattenText and strip_tags_full instead of creating duplicates (and so duplicate code), because that is just dirty ;).

Otherwise it looks fine. Please attach a new patch.

pnarula

pnarula

2011-06-20 18:01

reporter   ~15529

Hey Carsten,

I did so thinking that adding a new parameter to FlattenText() would affect the processing time as it is so extensively used. When I concatenated a string to $sNicetext, the page couldn't load and my computer hung up :( What's your opinion about this?

I've attached the patch though, now you have a choice ;)

c_schmitz

c_schmitz

2011-06-23 21:12

administrator   ~15570

The no_duplication patch looks fine. Please commit asap.
Thank you!

Mazi

Mazi

2011-11-11 16:45

updater   ~16652

According to http://www.limesurvey.org/en/forum/installation-a-update-issues/68968-issue-with-linebreaks-when-exporting-csv-file#68968 this doesn't seem to have been implemented?!

I consider this a useful feature so if it's no big deal it would be great to apply the patch to the next 1.91+ release.

c_schmitz

c_schmitz

2011-11-11 18:54

administrator   ~16656

Without further information I will have to close this issue because it works just fine here. Mazi, have you tested it?

sdondley

sdondley

2011-11-11 19:01

reporter   ~16657

OK, I looked at the raw csv and I do see line breaks. When I open it with the spreadsheet software, they get wiped out.

Can you tell me what program you are opening the csv file with so that the line breaks get preserved?

c_schmitz

c_schmitz

2011-11-11 19:17

administrator   ~16658

I am using LibreOffice Calc here.

sdondley

sdondley

2011-11-11 19:22

reporter   ~16659

OK, I'm running into this problem when I export using the MS Excel option. So I guess this isn't really related to the csv export.

c_schmitz

c_schmitz

2011-11-11 22:40

administrator   ~16661

SO, I can close the issue?

c_schmitz

c_schmitz

2011-11-15 16:25

administrator   ~16686

Closed due to missing feedback.

Related Changesets

LimeSurvey: Yii d19b8c15

2011-10-20 07:41:40

c_schmitz

Details Diff
Fixed issue 05126: CSV export - no new lines

git-svn-id: file:///Users/Shitiz/Downloads/lssvn/source/limesurvey_ci@11223 b72ed6b6-b9f8-46b5-92b4-906544132732
Affected Issues
05126
mod - application/helpers/admin/exportresults_helper.php Diff File
mod - application/helpers/common_helper.php Diff File

LimeSurvey: Yii 66df0776

2011-10-20 07:45:10

c_schmitz

Details Diff
Fixed issue 05126: CSV export - no new lines

git-svn-id: file:///Users/Shitiz/Downloads/lssvn/source/limesurvey_ci@11224 b72ed6b6-b9f8-46b5-92b4-906544132732
Affected Issues
05126
mod - application/helpers/admin/exportresults_helper.php Diff File

Issue History

Date Modified Username Field Change
2011-04-24 12:48 ngeorgiev New Issue
2011-04-24 21:35 c_schmitz Assigned To => pnarula
2011-04-24 21:35 c_schmitz Status new => assigned
2011-04-27 19:15 pnarula File Added: export_5126.patch
2011-04-27 19:19 pnarula Note Added: 14867
2011-04-28 09:54 pnarula Issue Monitored: pnarula
2011-04-28 09:54 pnarula Issue End Monitor: pnarula
2011-04-28 11:23 pnarula Issue Monitored: c_schmitz
2011-04-28 11:23 pnarula Note Added: 14869
2011-06-02 23:01 pnarula Assigned To pnarula => c_schmitz
2011-06-03 11:36 mot Note Added: 15270
2011-06-05 21:15 c_schmitz Note Added: 15303
2011-06-05 21:16 c_schmitz Assigned To c_schmitz => pnarula
2011-06-13 02:48 mot Issue Monitored: mot
2011-06-16 14:39 c_schmitz Note Added: 15465
2011-06-16 23:29 pnarula File Added: csv.fix.patch
2011-06-16 23:31 pnarula Note Added: 15470
2011-06-16 23:31 pnarula Assigned To pnarula => c_schmitz
2011-06-19 15:07 c_schmitz File Deleted: export_5126.patch
2011-06-19 15:09 c_schmitz Note Added: 15507
2011-06-19 16:47 c_schmitz Issue End Monitor: c_schmitz
2011-06-19 19:28 c_schmitz Assigned To c_schmitz => pnarula
2011-06-19 19:28 c_schmitz Status assigned => feedback
2011-06-20 16:16 pnarula File Added: csv_fix_stable_branch.patch
2011-06-20 16:17 pnarula Assigned To pnarula => c_schmitz
2011-06-20 16:36 c_schmitz Note Added: 15526
2011-06-20 16:37 c_schmitz Assigned To c_schmitz => pnarula
2011-06-20 16:37 c_schmitz Status feedback => assigned
2011-06-20 17:25 pnarula File Added: csv_fix_no_duplicate_code.patch
2011-06-20 17:26 pnarula File Deleted: csv_fix_no_duplicate_code.patch
2011-06-20 17:29 pnarula File Added: csv_fix_no_duplication.patch
2011-06-20 17:56 pnarula Assigned To pnarula =>
2011-06-20 18:01 pnarula Note Added: 15529
2011-06-20 18:01 pnarula Assigned To => c_schmitz
2011-06-23 21:11 c_schmitz Assigned To c_schmitz => pnarula
2011-06-23 21:12 c_schmitz Note Added: 15570
2011-06-23 21:40 pnarula Status assigned => resolved
2011-07-01 18:34 c_schmitz Status resolved => closed
2011-11-11 16:45 Mazi Assigned To pnarula => c_schmitz
2011-11-11 16:45 Mazi Note Added: 16652
2011-11-11 16:45 Mazi Status closed => feedback
2011-11-11 16:45 Mazi Resolution open => reopened
2011-11-11 18:54 c_schmitz Note Added: 16656
2011-11-11 19:01 sdondley Note Added: 16657
2011-11-11 19:17 c_schmitz Note Added: 16658
2011-11-11 19:22 sdondley Note Added: 16659
2011-11-11 22:40 c_schmitz Note Added: 16661
2011-11-15 16:25 c_schmitz Note Added: 16686
2011-11-15 16:25 c_schmitz Status feedback => closed
2011-11-15 16:25 c_schmitz Resolution reopened => unable to reproduce
2012-03-14 21:08 c_schmitz Changeset attached => Import 2012-03-09 13:30:34 Yii 66df0776
2012-03-14 21:08 c_schmitz Changeset attached => Import 2012-03-09 13:30:34 Yii d19b8c15
2021-08-03 00:43 guest Bug heat 10 => 12