View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
05126 | Bug reports | Import/Export | public | 2011-04-24 12:48 | 2012-03-14 21:08 |
Reporter | ngeorgiev | Assigned To | c_schmitz | ||
Priority | normal | Severity | minor | ||
Status | closed | Resolution | unable to reproduce | ||
Product Version | 1.91RC6 | ||||
Summary | 05126: CSV Export - no new lines | ||||
Description | I want to parse an exported CSV survey data, but it does not contain the new lines entered by the user. In my survey many users enter new lines and I need them for parsing. But they are not saved in the CSV file, so I have to add them manually (comparing the data with the notification email which contains the new lines). But it takes me at least 20 minutes for each survey entry! Can you add an option which allows the new lines to be saved in the exported CSV file? A recommendation from Mazi was to "add another filter option like "filter linebreaks". | ||||
Steps To Reproduce | . Make a survey with a question with long text | ||||
Additional Information | The bug is described also in the forum: | ||||
Tags | No tags attached. | ||||
Attached Files | csv.fix.patch (4,464 bytes)
Index: admin/export_data_functions.php =================================================================== --- admin/export_data_functions.php (revision 9992) +++ admin/export_data_functions.php (working copy) @@ -29,6 +29,21 @@ } /** + * Strips html tags + * + * @param $string + * @return $string + */ +function strip_tags_full_save_newline($string) { + $string=html_entity_decode($string, ENT_QUOTES, "UTF-8"); + //combining these into one mb_ereg_replace call ought to speed things up + //$string = str_replace(array("\r\n","\r","\n",'-oth-'), '', $string); + //The backslashes must be escaped twice, once for php, and again for the regexp + //$string = str_replace("'|\\\\'", "'", $string); + return FlattenTextWithNewline($string); +} + +/** * Returns true if passed $value is numeric * * @param $value @@ -46,9 +61,8 @@ return ($eng_or_world); } -function spss_export_data ($na = null) { +function spss_export_data ($na = null, $is_csv = false) { global $length_data; - // Build array that has to be returned $fields = spss_fieldmap(); @@ -138,9 +152,19 @@ echo("'0'"); } } elseif (!$field['hide']) { - $strTmp=mb_substr(strip_tags_full($row[$fieldno]), 0, $length_data); + if($is_csv == true) { + $strTmp=mb_substr(strip_tags_full_save_newline($row[$fieldno]), 0, $length_data); + } + else { + $strTmp=mb_substr(strip_tags_full($row[$fieldno]), 0, $length_data); + } if (trim($strTmp) != ''){ - $strTemp=str_replace(array("'","\n","\r"),array("''",' ',' '),trim($strTmp)); + if($is_csv == true) { + $strTemp=str_replace(array("'"),array("''"),trim($strTmp)); + } + else { + $strTemp=str_replace(array("'","\n","\r"),array("''",' ',' '),trim($strTmp)); + } /* * Temp quick fix for replacing decimal dots with comma's if (my_is_numeric($strTemp)) { Index: admin/export_data_r.php =================================================================== --- admin/export_data_r.php (revision 9992) +++ admin/export_data_r.php (working copy) @@ -138,7 +138,7 @@ header("Pragma: public"); $na=""; //change to empty string instead of two double quotes to fix warnings on NA - spss_export_data($na); + spss_export_data($na, true); // this true indicates that the format is .csv exit; } Index: common_functions.php =================================================================== --- common_functions.php (revision 9992) +++ common_functions.php (working copy) @@ -5008,7 +5008,7 @@ $sNicetext = strip_javascript($sTextToFlatten); $sNicetext = strip_tags($sNicetext); $sNicetext = str_replace(array("\n","\r"),array('',''), $sNicetext); - if ($bDecodeHTMLEntities==true) + if ($bDecodeHTMLEntities==true) { $sNicetext = str_replace(' ',' ', $sNicetext); // html_entity_decode does not properly convert to spaces $sNicetext = html_entity_decode($sNicetext, ENT_QUOTES, $sCharset); @@ -5019,6 +5019,30 @@ /** + * This functions removes all HTML tags, Javascript, and other strange chars from a given text + * + * @param string $sTextToFlatten Text you want to clean + * @param boolan $bDecodeHTMLEntities If set to true then all HTML entities will be decoded to the specified charset. Default: false + * @param string $sCharset Charset to decode to if $decodeHTMLEntities is set to true + * + * @return string Cleaned text + */ +function FlattenTextWithNewline($sTextToFlatten, $bDecodeHTMLEntities=false, $sCharset='UTF-8') +{ + $sNicetext = strip_javascript($sTextToFlatten); + $sNicetext = strip_tags($sNicetext); + $sNicetext = str_replace(array("\r\n","\r","\n"),array(PHP_EOL,PHP_EOL,PHP_EOL), $sNicetext); + if ($bDecodeHTMLEntities==true) + { + $sNicetext = str_replace(' ',' ', $sNicetext); // html_entity_decode does not properly convert to spaces + $sNicetext = html_entity_decode($sNicetext, ENT_QUOTES, $sCharset); + } + $sNicetext = trim($sNicetext); + return $sNicetext; +} + + +/** * getArrayFiltersForGroup() queries the database and produces a list of array_filter questions and targets with in the same group * @global string $surveyid * @global string $gid | ||||
Bug heat | 12 | ||||
Complete LimeSurvey version number (& build) | 9642 | ||||
I will donate to the project if issue is resolved | Yes | ||||
Browser | |||||
Database type & version | mysql | ||||
Server OS (if known) | Linux | ||||
Webserver software & version (if known) | unknown | ||||
PHP Version | should be more than 5 | ||||
Carsten, please have a look. |
|
Reminder sent to: c_schmitz Please review this patch.
Also do I need to assign the issue back to you or a reminder would be sufficient? |
|
I read it this way: Upon export to CSV, and option is given to filter linebreaks, like a checkbox [X] Filter Linebreaks If the option is ON, linebreaks will be filtered (removed/converted to space). If the option is OFF, linebreaks will be put into the CSV. So if some CSV readers don't support linebreaks when reading in CSV data, will not get broken. Ref: http://en.wikipedia.org/wiki/Comma-separated_values in specific:
So from a standards point of view, line breaks should be supported. But not by replacing them with {break} but just by properly writing the CSV output. Maybe what you suggest in the patch is a third way of doing so. For the moment I would say: 1.) The current implementation is broken as it does not follow the CSV specs. |
|
Pooja, I concur with mot's opinion. Can you create an according patch vs. stable please? |
|
? |
|
Uploaded the patch following the csv specifications. |
|
pnarula, the patch is vs. unstable. Please create a patch vs the stable branch. |
|
Thank you pnarula, can you change the patch by extending the existing functions like FlattenText and strip_tags_full instead of creating duplicates (and so duplicate code), because that is just dirty ;). Otherwise it looks fine. Please attach a new patch. |
|
Hey Carsten, I did so thinking that adding a new parameter to FlattenText() would affect the processing time as it is so extensively used. When I concatenated a string to $sNicetext, the page couldn't load and my computer hung up :( What's your opinion about this? I've attached the patch though, now you have a choice ;) |
|
The no_duplication patch looks fine. Please commit asap. |
|
According to http://www.limesurvey.org/en/forum/installation-a-update-issues/68968-issue-with-linebreaks-when-exporting-csv-file#68968 this doesn't seem to have been implemented?! I consider this a useful feature so if it's no big deal it would be great to apply the patch to the next 1.91+ release. |
|
Without further information I will have to close this issue because it works just fine here. Mazi, have you tested it? |
|
OK, I looked at the raw csv and I do see line breaks. When I open it with the spreadsheet software, they get wiped out. Can you tell me what program you are opening the csv file with so that the line breaks get preserved? |
|
I am using LibreOffice Calc here. |
|
OK, I'm running into this problem when I export using the MS Excel option. So I guess this isn't really related to the csv export. |
|
SO, I can close the issue? |
|
Closed due to missing feedback. |
|
LimeSurvey: Yii d19b8c15 2011-10-20 09:41 Details Diff |
Fixed issue 05126: CSV export - no new lines git-svn-id: file:///Users/Shitiz/Downloads/lssvn/source/limesurvey_ci@11223 b72ed6b6-b9f8-46b5-92b4-906544132732 |
Affected Issues 05126 |
|
mod - application/helpers/admin/exportresults_helper.php | Diff File | ||
mod - application/helpers/common_helper.php | Diff File | ||
LimeSurvey: Yii 66df0776 2011-10-20 09:45 Details Diff |
Fixed issue 05126: CSV export - no new lines git-svn-id: file:///Users/Shitiz/Downloads/lssvn/source/limesurvey_ci@11224 b72ed6b6-b9f8-46b5-92b4-906544132732 |
Affected Issues 05126 |
|
mod - application/helpers/admin/exportresults_helper.php | Diff File |
Date Modified | Username | Field | Change |
---|---|---|---|
2011-04-24 12:48 | ngeorgiev | New Issue | |
2011-04-24 21:35 | c_schmitz | Assigned To | => pnarula |
2011-04-24 21:35 | c_schmitz | Status | new => assigned |
2011-04-27 19:15 | pnarula | File Added: export_5126.patch | |
2011-04-27 19:19 | pnarula | Note Added: 14867 | |
2011-04-28 09:54 | pnarula | Issue Monitored: pnarula | |
2011-04-28 09:54 | pnarula | Issue End Monitor: pnarula | |
2011-04-28 11:23 | pnarula | Issue Monitored: c_schmitz | |
2011-04-28 11:23 | pnarula | Note Added: 14869 | |
2011-06-02 23:01 | pnarula | Assigned To | pnarula => c_schmitz |
2011-06-03 11:36 | mot | Note Added: 15270 | |
2011-06-05 21:15 | c_schmitz | Note Added: 15303 | |
2011-06-05 21:16 | c_schmitz | Assigned To | c_schmitz => pnarula |
2011-06-13 02:48 | mot | Issue Monitored: mot | |
2011-06-16 14:39 | c_schmitz | Note Added: 15465 | |
2011-06-16 23:29 | pnarula | File Added: csv.fix.patch | |
2011-06-16 23:31 | pnarula | Note Added: 15470 | |
2011-06-16 23:31 | pnarula | Assigned To | pnarula => c_schmitz |
2011-06-19 15:07 | c_schmitz | File Deleted: export_5126.patch | |
2011-06-19 15:09 | c_schmitz | Note Added: 15507 | |
2011-06-19 16:47 | c_schmitz | Issue End Monitor: c_schmitz | |
2011-06-19 19:28 | c_schmitz | Assigned To | c_schmitz => pnarula |
2011-06-19 19:28 | c_schmitz | Status | assigned => feedback |
2011-06-20 16:16 | pnarula | File Added: csv_fix_stable_branch.patch | |
2011-06-20 16:17 | pnarula | Assigned To | pnarula => c_schmitz |
2011-06-20 16:36 | c_schmitz | Note Added: 15526 | |
2011-06-20 16:37 | c_schmitz | Assigned To | c_schmitz => pnarula |
2011-06-20 16:37 | c_schmitz | Status | feedback => assigned |
2011-06-20 17:25 | pnarula | File Added: csv_fix_no_duplicate_code.patch | |
2011-06-20 17:26 | pnarula | File Deleted: csv_fix_no_duplicate_code.patch | |
2011-06-20 17:29 | pnarula | File Added: csv_fix_no_duplication.patch | |
2011-06-20 17:56 | pnarula | Assigned To | pnarula => |
2011-06-20 18:01 | pnarula | Note Added: 15529 | |
2011-06-20 18:01 | pnarula | Assigned To | => c_schmitz |
2011-06-23 21:11 | c_schmitz | Assigned To | c_schmitz => pnarula |
2011-06-23 21:12 | c_schmitz | Note Added: 15570 | |
2011-06-23 21:40 | pnarula | Status | assigned => resolved |
2011-07-01 18:34 | c_schmitz | Status | resolved => closed |
2011-11-11 16:45 | Mazi | Assigned To | pnarula => c_schmitz |
2011-11-11 16:45 | Mazi | Note Added: 16652 | |
2011-11-11 16:45 | Mazi | Status | closed => feedback |
2011-11-11 16:45 | Mazi | Resolution | open => reopened |
2011-11-11 18:54 | c_schmitz | Note Added: 16656 | |
2011-11-11 19:01 | sdondley | Note Added: 16657 | |
2011-11-11 19:17 | c_schmitz | Note Added: 16658 | |
2011-11-11 19:22 | sdondley | Note Added: 16659 | |
2011-11-11 22:40 | c_schmitz | Note Added: 16661 | |
2011-11-15 16:25 | c_schmitz | Note Added: 16686 | |
2011-11-15 16:25 | c_schmitz | Status | feedback => closed |
2011-11-15 16:25 | c_schmitz | Resolution | reopened => unable to reproduce |
2012-03-14 21:08 | c_schmitz | Changeset attached | => Import 2012-03-09 13:30:34 Yii 66df0776 |
2012-03-14 21:08 | c_schmitz | Changeset attached | => Import 2012-03-09 13:30:34 Yii d19b8c15 |
2021-08-03 00:43 | guest | Bug heat | 10 => 12 |