Relationship Graph
View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
17409 | Bug reports | Import/Export | public | 2021-06-28 10:15 | 2021-09-06 10:59 |
Reporter | Mazi | Assigned To | p_teichmann | ||
Priority | high | Severity | block | ||
Status | closed | Resolution | fixed | ||
Product Version | 4.4.12 | ||||
Summary | 17409: Exporting 3000 encrypted results (200 questions) to Excel requires too many ressources, export fails | ||||
Description | A custmer is running a 200 question survey and tries to export 3000 responses. This fals due to insufficient ressources (e.g. memory_limit setting). Exporting 50 data sets is doable but beyond that the export process takes too many ressources to complete and it fails with a 500 server error. Exporting that amount of ressources seems to be working fine if data is not enctpted so my guess is that the decryption is consuming too many ressources and needs to be optimized. Because 3000 responses at a medical o market research survey is not much. | ||||
Steps To Reproduce | Search for a test survey with many questions and many encrypted responses. It will probably be tricky to find the bottleneck. If data will be treated with care, the customer may be able to provide the DB and decryption keys for testing. Please let us know how we can provide that. | ||||
Tags | No tags attached. | ||||
Bug heat | 8 | ||||
Complete LimeSurvey version number (& build) | 4.4.12 | ||||
I will donate to the project if issue is resolved | No | ||||
Browser | |||||
Database type & version | mysql Ver 15.1 Distrib 10.2.36-MariaDB | ||||
Server OS (if known) | SUSE SLES 15 SP 2 | ||||
Webserver software & version (if known) | |||||
PHP Version | 7.4.6 | ||||
related to | 17126 | closed | p_teichmann | encrypted questions do not work when multiple groups are used |
How low? |
|
php.ini check for setting "memory_limit" and try e.g. 128M or just 64M |
|
OK, but is that a problem? 512M is a pretty normal setting by now. |
|
Question is if we can handle 800 questions with 100k answers. We need a proper stress-test to figure out the limits of the system. |
|
So possible subtask: Create a command to generate random responses for any survey. |
|
They have copied the DB to a new laptop and have significantly raised the limits in order to export the data. I think even with 1024M they falied exporting the 3000 data sets. I have asked them to prepare a DB dump and they will make it available for download later. I will email you and Patrick the download links once available. Of course, you can not support super-large data sets if there are no sufficient ressources, but still with 1024M they can not export 3000 responses which is not much. > 20% of our projects have more responses and >5-10% come with > 10000 data sets. So we need to optimize the export if possible. |
|
Yes, that's not good. :) Thanks for reporting. I did nag the original implementer (Dominik) to stress test it in different ways, but he quit before. |
|
@ollehar, please test the export with the DB details provided by the customer. Can you reproduce? |
|
We have already reproduced and fixed it, its currently being tested |
|
Happy to hear that, @p_teichmann. Looking forward to see the fix at the next release so we can test that one. |
|
@p_teichmann, it looks like the fix is not yet available at yesterday's release (accoridng to https://github.com/LimeSurvey/LimeSurvey/blob/master/docs/release_notes.txt)? What a pity, we really need this to export data of our current project! I also couldn't find a potential fix at the development branch so I assume this was fixed locally only? Would it be possible to provide the fix so we can test as well? |
|
Potential fix is available at https://github.com/LimeSurvey/LimeSurvey/commit/b5cac3393e06d6b828e55fafd386322b83e58507 |
|
Still working on QA test... |
|
@Mazi Did you test this fix? It's branch bug/L41-T891_Exporting_3000_encryptedresults(200_questions)_to_Excel_requires_too_many_ressources,_export_fails, as you linked. |
|
@ollehar, we tried but with no success. But the reaosn is not the fix I guess but the bug at https://bugs.limesurvey.org/view.php?id=17126 having caused the data to be encrypted multi times. |
|
Ok? We have a database dump from you we're trying to test with. Did you use the same one? |
|
@ollehar, please contact @p_teichmann directly, he already worked on this with the details we provided. |
|
Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=32530 |
|
LimeSurvey: master b72beace 2021-07-02 14:14:07 Details Diff |
Fixed issue 17409: Exporting responses with big data sets and activated encryption will timeout php memory limit, even with high settings (512M) |
Affected Issues 17409 |
|
mod - application/helpers/admin/export/SurveyDao.php | Diff File | ||
mod - application/helpers/admin/exportresults_helper.php | Diff File |
Date Modified | Username | Field | Change |
---|---|---|---|
2021-06-28 10:15 | Mazi | New Issue | |
2021-06-28 10:35 | ollehar | Priority | none => urgent |
2021-06-28 10:35 | ollehar | Sync to Zoho Project | => |Yes| |
2021-06-28 10:40 | ollehar | Assigned To | => p_teichmann |
2021-06-28 10:40 | ollehar | Status | new => assigned |
2021-06-28 10:41 | ollehar | Note Added: 65113 | |
2021-06-28 10:41 | ollehar | Status | assigned => feedback |
2021-06-28 10:52 | Mazi | Note Added: 65114 | |
2021-06-28 10:52 | Mazi | Status | feedback => assigned |
2021-06-28 11:27 | ollehar | Note Added: 65115 | |
2021-06-28 11:27 | ollehar | Note Added: 65116 | |
2021-06-28 11:35 | ollehar | Note Added: 65117 | |
2021-06-28 11:39 | Mazi | Note Added: 65118 | |
2021-06-28 11:41 | ollehar | Note Added: 65119 | |
2021-07-06 11:11 | Mazi | Note Added: 65228 | |
2021-07-06 12:53 | p_teichmann | Note Added: 65231 | |
2021-07-07 16:40 | Mazi | Note Added: 65247 | |
2021-07-13 23:23 | Mazi | Note Added: 65433 | |
2021-07-14 00:42 | Mazi | Note Added: 65434 | |
2021-07-14 13:16 | Mazi | Relationship added | related to 17126 |
2021-07-21 16:10 | ollehar | Note Added: 65536 | |
2021-07-21 16:13 | ollehar | Note Added: 65537 | |
2021-07-21 16:18 | ollehar | Status | assigned => ready for testing |
2021-07-21 19:53 | Mazi | Note Added: 65563 | |
2021-07-22 08:21 | ollehar | Note Added: 65564 | |
2021-07-22 13:01 | galads | Issue Monitored: galads | |
2021-07-24 23:05 | Mazi | Note Added: 65605 | |
2021-08-04 00:08 | ollehar | Priority | urgent => high |
2021-08-04 00:08 | ollehar | Sync to Zoho Project | Yes => |Yes| |
2021-08-13 08:29 | p_teichmann | Changeset attached | => LimeSurvey master b72beace |
2021-08-13 08:29 | p_teichmann | Note Added: 65957 | |
2021-08-13 08:29 | p_teichmann | Resolution | open => fixed |
2021-08-30 14:01 | galads | Status | ready for testing => resolved |
2021-09-06 10:59 | c_schmitz | Status | resolved => closed |