View Issue Details

This bug affects 1 person(s).
 14
IDProjectCategoryView StatusLast Update
06567Bug reportsCentral participant databasepublic2012-11-15 14:47
Reporterlolo Assigned Tomdekker  
PrioritynormalSeverityminor 
Status closedResolutionfixed 
Product Version2.00RC9 
Fixed in Version2.00+ 
Summary06567: Performance issue
Description

My participant database currently has:

  • 440 participants
  • 37 attributes

On the panel screen, to display the full list of participants it takes:

  • 31 seconds on my hosted provider
  • 11 seconds on my local test server (Intel Core 2 Duo 3Ghz, 4GB RAM)

There has to be a bootleneck somewhere, as it's not that much data to handle even for modest hardware.

Steps To Reproduce
  • Populate the cpdb with similar amount of data
  • log in LS
  • go to Central Participant Database/Panel
  • click "Display participants" icon
TagsNo tags attached.
Bug heat14
Complete LimeSurvey version number (& build)120817
I will donate to the project if issue is resolvedNo
BrowserMozilla Firefox 15.0.1
Database type & versionMysql 5.5.24
Server OS (if known)Linux 2.6.27.52 & Win XP SP3
Webserver software & version (if known)Apache 2
PHP Version5.3.14

Users monitoring this issue

aniesshsethh, lolo

Activities

lolo

lolo

2012-09-19 12:12

reporter   ~20729

Forgot last step in "Steps to reproduce":

  • set page size to display all entries
aniesshsethh

aniesshsethh

2012-10-28 00:32

reporter   ~21736

Fix committed to 2.1 branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=10037

aniesshsethh

aniesshsethh

2012-10-28 01:02

reporter   ~21737

Last edited: 2012-10-28 01:13

Hey Lolo,

I did some testing with 3749 participants and 26 attributes, and yes there was a bottleneck that has been fixed and committed.

I want to share the test results so as to bring something to your notice. Initially these participants without any attribute visible in the display participant screen were taking 19.4 seconds to load. After removing the bottleneck it came down to 2.25 seconds

Then I enabled all the 26 attributes to be visible in the display participant screen , the load time was 30 seconds and with 6 attributes it was 9.12 seconds and with 1 attribute visible it was 4 seconds.

I would suggest that you don't keep the unnecessary attributes visible in the participant screen. Even if the attribute value is empty, that will still have an impact on performance. Keep the attributes visible to only the ones you require :)

Mazi

Mazi

2012-10-28 14:12

updater   ~21738

Aniessh, for a real panel provider, having 20 or more attributes is absolutely normal. So if +1 attribute adds +1sec loading time, there still seems to be some kind of bottleneck I would guess. Maybe you can have another look before more bug report about perfomance issues with >10 attributes show up?

lolo

lolo

2012-10-28 16:39

reporter   ~21741

Hi Aniessh,
Thanks for looking into this (my system is becoming less responsive every day as the panel is growing).
Regarding the bottleneck, it seems -in my humble opinion- to be a scalability issue.

If panel contains N participants and M attributes, I suspect that the application is performing N x M requests (HTTP+database). This is never going to scale with large amounts of data, as the performance penalty is exponential.

The solution, in my opinion, is to aim for an algorithm of linear complexity (or even better, a constant complexity).

The best solution would be to fetch all participants and their respective attributes in one single SQL query (and therefore one single HTTP/json request). In issue 06454 I gave an example of such a SQL query.

On my MySQL server I have created a view that contains such a query (it returns all participants and all their attribute values in one single SQL query, so I know it works).

I know it's easier said than done, and that it would require quite a bit of changes, but it may be the only way to definitely solve the issue.

Hope it helped.

aniesshsethh

aniesshsethh

2012-10-30 16:21

reporter   ~21780

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=10056

aniesshsethh

aniesshsethh

2012-10-30 19:45

reporter   ~21796

Fix committed to 2.1 branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=10066

aniesshsethh

aniesshsethh

2012-10-30 19:58

reporter   ~21797

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=10068

aniesshsethh

aniesshsethh

2012-10-30 20:07

reporter   ~21798

Last edited: 2012-10-30 20:16

Hey Mazi,Lolo

@Mazi : sure :)

@lolo : thank you very much for the example of the query you gave, I implemented that query and it's working like a charm. Please note I have done this for users with super admin rights only, will try to do for normal users tomorrow ( feeling sleepy ) ,it's quite different from super admin. Although I have one question for you, if you can answer that then we can probably reduce the computation time further, do you have any idea of how to handle the survey count in that single query, I mean it's a join on the number of counts on survey_links table, if that's possible and if you can give an example of the same, I would be more than happy to implement that and the entire data will then come from a single query which will reduce the turn around time even further.

As far as the number of http/json requests is concerned, it's has always been a single request to the server, but because of the execution of large number of queries it was running slow.

As for statistics , It was taking around 31.33 with 26 attributes and 3749 participants which is now reduced to 4.08 seconds with the same number of attributes, thanks to your example of the query :)

c_schmitz

c_schmitz

2012-11-07 16:28

administrator   ~22001

The fix contains hard-coded table prefixes, see https://github.com/LimeSurvey/LimeSurvey/commit/3c6ea940e6c2d04fccbc33056ab61dd4e5c6bb70#L1R215

Aniessh, can you please fix this?

mdekker

mdekker

2012-11-07 16:59

reporter   ~22002

Added some fixes to participant panel, including the prefix issue

lolo

lolo

2012-11-15 14:24

reporter   ~22181

Last edited: 2012-11-15 14:25

Hi,

Recently updated to build 121115, and the rendering performance is now miles away than what it was before.

So, congrats for the speed-ups guys, it's much more usable now !

Related Changesets

LimeSurvey: 2.1 c2e438b0

2012-10-27 15:23:42

aniesshsethh

Details Diff
Fixed issue 06567: Performance issue
Dev Removed unnecessary processing giving display participant better load times
Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File

LimeSurvey: master 7a08f153

2012-10-30 08:20:45

aniesshsethh

Details Diff
Fixed issue 06567: Performance issue
Dev Removed unnecessary processing giving display participant better load times
Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File

LimeSurvey: 2.1 b8a1073c

2012-10-30 11:44:14

aniesshsethh

Details Diff
Fixed issue 06567: Performance issue
Dev This commit changes the way attributes are extracted for superadmins only
Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File
mod - application/models/Participants.php Diff File

LimeSurvey: master 3c6ea940

2012-10-30 11:58:15

aniesshsethh

Details Diff
Fixed issue 06567: Performance issue
Dev This commit changes the way attributes are extracted for superadmins only
Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File
mod - application/models/Participants.php Diff File

LimeSurvey: master a88ebb5c

2012-11-07 15:45:02

mdekker

Details Diff
dev: minor fixes to participant panel Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File
mod - application/models/Participants.php Diff File

LimeSurvey: master 65e4d015

2012-11-07 15:48:25

mdekker

Details Diff
dev: removed debug & reset Affected Issues
06567
mod - application/controllers/admin/participantsaction.php Diff File

LimeSurvey: master 670eaaa8

2012-11-07 15:52:39

mdekker

Details Diff
dev: another small fix Affected Issues
06567
mod - application/models/ParticipantAttributeNames.php Diff File

Issue History

Date Modified Username Field Change
2012-09-19 11:58 lolo New Issue
2012-09-19 11:58 lolo Status new => assigned
2012-09-19 11:58 lolo Assigned To => jcleeland
2012-09-19 12:12 lolo Note Added: 20729
2012-09-19 12:13 lolo Issue Monitored: lolo
2012-10-14 13:42 Mazi Issue Monitored: aniesshsethh
2012-10-28 00:32 aniesshsethh Changeset attached => LimeSurvey 2.1 c2e438b0
2012-10-28 00:32 aniesshsethh Note Added: 21736
2012-10-28 00:32 aniesshsethh Assigned To jcleeland => aniesshsethh
2012-10-28 00:32 aniesshsethh Resolution open => fixed
2012-10-28 01:02 aniesshsethh Note Added: 21737
2012-10-28 01:02 aniesshsethh Status assigned => feedback
2012-10-28 01:05 aniesshsethh Note Edited: 21737
2012-10-28 01:13 aniesshsethh Note Edited: 21737
2012-10-28 14:12 Mazi Note Added: 21738
2012-10-28 16:39 lolo Note Added: 21741
2012-10-28 16:39 lolo Status feedback => assigned
2012-10-28 21:44 rizaemet Issue Monitored: rizaemet
2012-10-28 21:45 rizaemet Issue End Monitor: rizaemet
2012-10-30 16:21 aniesshsethh Changeset attached => LimeSurvey master 7a08f153
2012-10-30 16:21 aniesshsethh Note Added: 21780
2012-10-30 19:45 aniesshsethh Changeset attached => LimeSurvey 2.1 b8a1073c
2012-10-30 19:45 aniesshsethh Note Added: 21796
2012-10-30 19:58 aniesshsethh Changeset attached => LimeSurvey master 3c6ea940
2012-10-30 19:58 aniesshsethh Note Added: 21797
2012-10-30 20:07 aniesshsethh Note Added: 21798
2012-10-30 20:07 aniesshsethh Status assigned => feedback
2012-10-30 20:12 aniesshsethh Note Edited: 21798
2012-10-30 20:15 aniesshsethh Note Edited: 21798
2012-10-30 20:16 aniesshsethh Note Edited: 21798
2012-11-06 10:24 c_schmitz Status feedback => closed
2012-11-06 10:25 c_schmitz Fixed in Version => 2.00+
2012-11-07 16:28 c_schmitz Note Added: 22001
2012-11-07 16:28 c_schmitz Status closed => feedback
2012-11-07 16:28 c_schmitz Resolution fixed => reopened
2012-11-07 16:58 mdekker Changeset attached => LimeSurvey master a88ebb5c
2012-11-07 16:59 mdekker Changeset attached => LimeSurvey master 65e4d015
2012-11-07 16:59 mdekker Changeset attached => LimeSurvey master 670eaaa8
2012-11-07 16:59 mdekker Note Added: 22002
2012-11-15 14:24 lolo Note Added: 22181
2012-11-15 14:24 lolo Status feedback => assigned
2012-11-15 14:25 lolo Note Edited: 22181
2012-11-15 14:47 c_schmitz Status assigned => closed
2012-11-15 14:47 c_schmitz Assigned To aniesshsethh => mdekker
2012-11-15 14:47 c_schmitz Resolution reopened => fixed
2021-08-02 19:35 guest Bug heat 10 => 14