View Issue Details

This issue affects 1 person(s).
 8
IDProjectCategoryView StatusLast Update
19452Bug reportsOtherpublic2024-04-02 14:01
Reportertibor.pacalat Assigned Tolajosarpad 
PrioritynoneSeveritytweak 
Status closedResolutionfixed 
Product Version6.4.x 
Summary19452: Improve usage of getQuestionAttributes
Description

This is a follow-up of another ticket where we have found a simple way to improve performance overall, as can be seen at the PR
https://github.com/LimeSurvey/LimeSurvey/pull/3751/files

Short description of the improvement:

Earlier the method only supported to get integers representing a question through its ID, querying the question and caching it for the next time. However, when I tested this with a survey having 600+ questions and clicked on any of the left menu tabs, the method was called separately for all questions, leading to 1 + 600+ queries to the database, the first query loading all questions together, then looping the questions, passing their id and loading the question from database despite the fact that it was already loaded.

The solution to that problem was to allow this method to receive a question object instead of an id and consequently use that object and cache it rather than loading it again, reducing the number of queries for questions in this context from 1 + n to 1 and having this magnitude of speed improvement when the survey had more than 600 questions (image1).

What we need in order to further improve is to apply this idea to other usages of the method, with the pattern of the action we desire to do looking like this:

  1. look at each reference
    a) try to find the question separately and with as few DB requests as possible (preferrably 1)
    b) pass the object to the method
    c) test whether it works well, whether it improves performance and what other effects the improvement has
  2. change the method to only accept the question object if possible
  3. wherever possible : load the needed question objects via a single query

So, we will need to review and refactor the usages of this method with the aim of:
a) improving performance
b) stopping support for integer being passed to it
c) making the code more compact, readable and understandable

Maybe we could create a PR for each of the files referring this method at:

  • application/controllers/ResponsesController.php (done, PR https://github.com/LimeSurvey/LimeSurvey/pull/3769)
  • application/controllers/UploaderController.php (done, PR https://github.com/LimeSurvey/LimeSurvey/pull/3782)
  • application/controllers/admin/ConditionsAction.php (done, PR https://github.com/LimeSurvey/LimeSurvey/pull/3783)
  • application/controllers/admin/DataEntry.php (PR https://github.com/LimeSurvey/LimeSurvey/pull/3784)
  • application/controllers/admin/PrintableSurvey.php (PR https://github.com/LimeSurvey/LimeSurvey/pull/3789)
  • application/controllers/admin/Statistics.php (PR https://github.com/LimeSurvey/LimeSurvey/pull/3793)
  • application/core/LS_Twig_Extension.php (Á)
  • application/core/plugins/ExportSPSSsav/SPSSWriter.php
  • application/helpers/common_helper.php
  • application/helpers/export_helper.php
  • application/helpers/frontend_helper.php
  • application/helpers/qanda_helper.php
  • application/helpers/SurveyRuntimeHelper.php
  • application/helpers/userstatistics_helper.php
  • application/helpers/admin/import_helper.php
  • application/helpers/admin/statistics_helper.php
  • application/helpers/admin/export/SurveyObj.php
  • application/helpers/expressions/em_manager_helper.php
  • application/helpers/remotecontrol/remotecontrol_handle.php
  • application/libraries/Api/Command/V1/Transformer/Output/TransformerOutputSurveyDetail.php
  • application/libraries/PluginManager/LimesurveyApi.php
  • application/models/Question.php
  • application/models/QuestionAttribute.php
  • application/models/QuestionBaseRenderer.php
  • application/models/QuestionTemplate.php
  • application/models/SurveyDynamic.php
  • application/models/services/CoreQuestionAttributeProvider.php
  • application/models/services/QuestionAggregateService/QuestionService.php
  • application/views/admin/export/statistics_subviews/_question.php

Further research needed in order to improve performance, make the server more reliable and scalable:

  • check for any uses of getQuestionAttributes in the cloud version that’s not in the community edition and create a new ticket and a similar list of things to be improved
  • check for this pattern of individually loading the same kind of entities individually rather than loading them together for other important entities, such as:
    a) surveys
    b) answers
    c) whatever else we can think of

This ticket is also available in JIRA https://limesurvey.atlassian.net/browse/MW-1040, but please keep the communication about it here, so we have everything in the same place.

Steps To Reproduce

Steps to reproduce

(Replace this text with detailed step-by-step instructions on how to reproduce the issue)

Expected result

(Write here what you expected to happen)

Actual result

(Write here what happened instead)

Additional Information

Add 3. wherever possible : load the needed question objects via a single query

TagsNo tags attached.
Attached Files
image1.png (39,948 bytes)   
image1.png (39,948 bytes)   
Bug heat8
Complete LimeSurvey version number (& build)6.4.9+240226
I will donate to the project if issue is resolvedNo
Browser
Database type & version.
Server OS (if known)
Webserver software & version (if known)
PHP Version.
Git branch / Pull Request URLN/A

Users monitoring this issue

There are no users monitoring this issue.

Activities

tibor.pacalat

tibor.pacalat

2024-03-01 13:21

administrator   ~79700

Hi @DenisChenu @gabrieljenik! This is the ticket I talked about.
Please go file by file and make separate PRs for each.
After you are done with the first PR, please assign it to Arpad for code review and then if everything is fine continue working on this.
When you take a file to work on, please add your name in the ticket description next to it in brackets, so that we don't step on each other's toes.

Thank you very much in advance!

DenisChenu

DenisChenu

2024-03-04 08:12

developer   ~79702

After you are done with the first PR, please assign it to Arpad for code review and then if everything is fine continue working on this.

Maybe start by a PR with existing Question objetc (we already have/use Question object on the function)

tibor.pacalat

tibor.pacalat

2024-03-04 10:50

administrator   ~79704

Maybe start by a PR with existing Question objetc (we already have/use Question object on the function)

I do not understand what you mean. Can you please clarify?

DenisChenu

DenisChenu

2024-03-04 12:31

developer   ~79705

I do not understand what you mean. Can you please clarify?

It's more organizational : check if we use getQuestionAttributes($oQuestioon->qid) but : right : seem more easy with file by file PR …

tibor.pacalat

tibor.pacalat

2024-03-04 13:06

administrator   ~79708

Yes, please do it like asked in the ticket. Thank you!

DenisChenu

DenisChenu

2024-03-04 15:23

developer   ~79711

Bug or dev ?
Not really a bug : i move the issue to Development (and don't add the fixed issue in commit)

DenisChenu

DenisChenu

2024-03-06 16:51

developer   ~79748

application/controllers/UploaderController.php https://github.com/LimeSurvey/LimeSurvey/pull/3782

DenisChenu

DenisChenu

2024-03-06 17:03

developer   ~79749

application/controllers/admin/ConditionsAction.php https://github.com/LimeSurvey/LimeSurvey/pull/3783

DenisChenu

DenisChenu

2024-03-13 14:36

developer   ~79788

I stop on this one

DenisChenu

DenisChenu

2024-03-13 14:42

developer   ~79789

I stop because: it's not restricted, it's "Fix all possible issue that can happen on this controller"

DenisChenu

DenisChenu

2024-03-21 13:48

developer   ~79816

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=36228

DenisChenu

DenisChenu

2024-03-21 13:48

developer   ~79817

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=36227

DenisChenu

DenisChenu

2024-03-21 14:44

developer   ~79818

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=36229

DenisChenu

DenisChenu

2024-03-21 14:44

developer   ~79819

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=36230

DenisChenu

DenisChenu

2024-03-21 14:49

developer   ~79820

Fix committed to master branch: http://bugs.limesurvey.org/plugin.php?page=Source/view&id=36231

DenisChenu

DenisChenu

2024-03-26 08:47

developer   ~79828

I stop because: it's not restricted, it's "Fix all possible issue that can happen on this controller"

I really don't want to work on other controller. I really think it's a really bad solution to use a commit to fix other issue's.

And here : without analysis the source of the issue (like ExpressionManager without question status …)

lajosarpad

lajosarpad

2024-03-26 12:38

reporter   ~79830

"I stop because: it's not restricted, it's "Fix all possible issue that can happen on this controller""

This statement is false. I did not intend to delve into long discussions with you about this ticket, because we both have unnecessarily invested considerable amount of time into discussing things, like clarifying that we actually need to improve upon performance, which you inaccurately claimed that was not clarified in the ticket from the start.

But your claim that we fix all issues that can happen in a controller, controllers being of thousands of lines unrelated to getQuestionAttributes is untrue, we only fix issues that make our changes not operational, independently of whether they were new or old

We focus on improving upon the usages of a given method. We intend to change how we use it, which includes:

  1. Improvements upon the parameter usage, so that we no longer pass numeric ids to the method, but Question objects itself, with the goal of improving performance.

  2. We improve performance wherever it's possible by not loading the Questions individually like you did for quite a while. The main aim of this work is to improve upon speed, which we clarified already in the initial specification, even though you claimed later that we didn't.

  3. We polish the code we are touching, if there is obvious improvement that we can make, then we do not avoid doing it just because we did not specify it. We cannot foresee all the minor and major issues that we may encounter, but if we never fix or improve anything that was not specifically mentioned, then some faults of the code will never improve. However, this polishing is generally optional.

  4. If the code is not operational, if it crashes then we need to fix the reason of the crash. We cannot create many tickets for all reasons for the code crashing, like "fix the issue at line xyz for PR 456", because if there are many such issues, then we will wait for months for a PR to be mergeable and we only see the second issue once we passed the first. We intend to switch from Limesurvey5 to Limesurvey6 and we cannot do it while Limesurvey6 is slow. So all the superfluous new tickets that you propose to create as new prerequisites for these PRs (instead of just fixing the cause of the crash) would have no other result than postponing the time when the upgrade happens.

DenisChenu

DenisChenu

2024-03-26 12:53

developer   ~79831

It's not related to the way we get Question object i'm OK here.

But issues with survey that are totally unrelated
https://github.com/LimeSurvey/LimeSurvey/pull/3789#discussion_r1522656350
https://github.com/LimeSurvey/LimeSurvey/pull/3789#discussion_r1522984142
https://github.com/LimeSurvey/LimeSurvey/pull/3789#discussion_r1522662524 (the worst here : it show potential issue with ExpressionManager)

I just don't want to work on this commit , it's OK ?

lajosarpad

lajosarpad

2024-03-26 14:18

reporter   ~79834

No worries, we can handle it.

LimeBot

LimeBot

2024-04-02 14:01

administrator   ~79857

Fixed in Release 6.5.2+240402

Related Changesets

LimeSurvey: master 17af24ca

2024-03-21 14:48

DenisChenu

Committer: GitHub


Details Diff
Fixed issue 19452 : getQuestionAttributes for UploaderController

* Dev: 19452 : getQuestionAttributes for UploaderController
Affected Issues
19452
mod - application/controllers/UploaderController.php Diff File

LimeSurvey: master 17af24ca

2024-03-21 14:48

DenisChenu

Committer: GitHub


Details Diff
Fixed issue 19452 : getQuestionAttributes for UploaderController

* Dev: 19452 : getQuestionAttributes for UploaderController
Affected Issues
19452
mod - application/controllers/UploaderController.php Diff File

LimeSurvey: master 1f846862

2024-03-21 15:24

DenisChenu

Committer: GitHub


Details Diff
Fixed issue 19452: Improve usage of getQuestionAttributes DataEntry (03784)

* Dev: 19452: Improve usage of getQuestionAttributes DataEntry

* Dev: more optimisation

* Dev: Re add QuestionAttribute::model()->getQuestionAttributes(Question::model()->findByPk( when it's neeed
Dev: not for other question, and not in other foreach

* MW-1040: Improved upon usages of getQuestionAttributes at DataEntry

* MW-1040: DataEntry improvements

---------

Co-authored-by: Lajos Arpad <arpad@endpoint.com>
Affected Issues
19452
mod - application/controllers/admin/DataEntry.php Diff File

LimeSurvey: master 1f846862

2024-03-21 15:24

DenisChenu

Committer: GitHub


Details Diff
Fixed issue 19452: Improve usage of getQuestionAttributes DataEntry (03784)

* Dev: 19452: Improve usage of getQuestionAttributes DataEntry

* Dev: more optimisation

* Dev: Re add QuestionAttribute::model()->getQuestionAttributes(Question::model()->findByPk( when it's neeed
Dev: not for other question, and not in other foreach

* MW-1040: Improved upon usages of getQuestionAttributes at DataEntry

* MW-1040: DataEntry improvements

---------

Co-authored-by: Lajos Arpad <arpad@endpoint.com>
Affected Issues
19452
mod - application/controllers/admin/DataEntry.php Diff File

LimeSurvey: master 570b810f

2024-03-21 15:31

DenisChenu

Committer: GitHub


Details Diff
Fixed issue 19452: Improve usage of getQuestionAttributes PrintableSurvey (03789)

* Dev 19452: Improve usage of getQuestionAttributes PrintableSurvey
Dev: simple for question
Dev: add specific array for conditions (again array dual scale)

* Dev: avoid multiple call of same question attributes

* MW-1040: Applied some fixes

* Added a comment

* MW-1040: Fixed a typo

---------

Co-authored-by: Lajos Arpad <arpad@endpoint.com>
Affected Issues
19452
mod - application/controllers/admin/PrintableSurvey.php Diff File

Issue History

Date Modified Username Field Change
2024-03-01 13:16 tibor.pacalat New Issue
2024-03-01 13:16 tibor.pacalat File Added: image1.png
2024-03-01 13:16 tibor.pacalat Assigned To => DenisChenu
2024-03-01 13:16 tibor.pacalat Status new => assigned
2024-03-01 13:21 tibor.pacalat Note Added: 79700
2024-03-01 13:21 tibor.pacalat Bug heat 0 => 2
2024-03-04 08:12 DenisChenu Note Added: 79702
2024-03-04 08:12 DenisChenu Bug heat 2 => 4
2024-03-04 10:50 tibor.pacalat Note Added: 79704
2024-03-04 12:31 DenisChenu Note Added: 79705
2024-03-04 13:06 tibor.pacalat Note Added: 79708
2024-03-04 15:23 DenisChenu Note Added: 79711
2024-03-04 15:23 DenisChenu Project Bug reports => @10@
2024-03-06 16:51 DenisChenu Note Added: 79748
2024-03-06 17:03 DenisChenu Note Added: 79749
2024-03-11 15:25 lajosarpad Description Updated
2024-03-11 19:19 lajosarpad Description Updated
2024-03-12 08:49 DenisChenu Description Updated
2024-03-12 08:49 DenisChenu Additional Information Updated
2024-03-12 18:10 DenisChenu Description Updated
2024-03-13 14:36 DenisChenu Note Added: 79788
2024-03-13 14:36 DenisChenu Assigned To DenisChenu => tibor.pacalat
2024-03-13 14:42 DenisChenu Note Added: 79789
2024-03-19 19:14 lajosarpad Description Updated
2024-03-19 22:10 lajosarpad Description Updated
2024-03-20 15:01 lajosarpad Description Updated
2024-03-21 13:48 DenisChenu Changeset attached => LimeSurvey master 17af24ca
2024-03-21 13:48 DenisChenu Changeset attached => LimeSurvey master 17af24ca
2024-03-21 13:48 DenisChenu Note Added: 79816
2024-03-21 13:48 DenisChenu Note Added: 79817
2024-03-21 13:48 DenisChenu Assigned To tibor.pacalat => DenisChenu
2024-03-21 13:48 DenisChenu Assigned To tibor.pacalat => DenisChenu
2024-03-21 13:48 DenisChenu Resolution open => fixed
2024-03-21 13:48 DenisChenu Resolution open => fixed
2024-03-21 14:44 DenisChenu Changeset attached => LimeSurvey master 1f846862
2024-03-21 14:44 DenisChenu Changeset attached => LimeSurvey master 1f846862
2024-03-21 14:44 DenisChenu Note Added: 79818
2024-03-21 14:44 DenisChenu Note Added: 79819
2024-03-21 14:49 DenisChenu Changeset attached => LimeSurvey master 570b810f
2024-03-21 14:49 DenisChenu Note Added: 79820
2024-03-26 08:46 DenisChenu Assigned To DenisChenu => lajosarpad
2024-03-26 08:47 DenisChenu Note Added: 79828
2024-03-26 12:38 lajosarpad Note Added: 79830
2024-03-26 12:38 lajosarpad Bug heat 4 => 6
2024-03-26 12:53 DenisChenu Note Added: 79831
2024-03-26 14:18 lajosarpad Note Added: 79834
2024-04-02 14:01 LimeBot Note Added: 79857
2024-04-02 14:01 LimeBot Status assigned => closed
2024-04-02 14:01 LimeBot Bug heat 6 => 8