View Issue Details

IDProjectCategoryView StatusLast Update
14859Feature requestsSurvey takingpublic2019-12-18 12:32
Reporterollehar Assigned Toollehar  
PrioritynoneSeverityfeature 
Status assignedResolutionopen 
Summary14859: Cache result from createFieldmap etc
Description

How much can be cached and reused in the front-end loading of LS? Can we save the result from createFieldmap in file or database to be reused for the next participant?

Additional Information

Spec here: https://project.limesurvey.org/T264

TagsNo tags attached.

Activities

ollehar

ollehar

2019-05-07 20:10

administrator   ~51816

@DenisChenu Thoughts?

DenisChenu

DenisChenu

2019-05-08 10:31

developer   ~51817

I think i already put a Dev part about this. And it's a really good idea.

The point :

  1. We must clean/disable cache when admin user update survey : then create a new function
  2. Current createFielsMap get : url params + random + token value : must separate
  3. We must do same things for some EM function https://github.com/LimeSurvey/LimeSurvey/blob/ffee4e19f81f9d0130fb033d394d6d8540ba0fcc/application/helpers/expressions/em_manager_helper.php#L4987 maybe ?
ollehar

ollehar

2019-05-08 10:54

administrator   ~51818

  1. Yep, saving anything will mark the cache as dirty. The cache is only active when the survey is activated.
  2. True, some parts are individual and need to be overwritten.
  3. I did a benchmark of LS - EM run preg_match 180k times during one survey load! (Demo survey.)
DenisChenu

DenisChenu

2019-05-08 11:01

developer   ~51819

  1. Not only : current EM session contains question text : i think we must remove it from session.

Maybe we must remove some part for

  1. Session (to be moved to cache only)
  2. Cache (to be moved to DB only)
  3. Session only (Token value, current answers value, prefill url value …), maybe separate session : survey_XXXX and livesurvey_XXX

For example : i‘m unsure we need question.text : i think it's to have it in js expression : but it false : https://bugs.limesurvey.org/view.php?id=14817 , then we can, call it in EM from DB when needed (One DB line call : quick).

PS : i really think it's a REALY complex system and come with a lot of issues …

DenisChenu

DenisChenu

2019-05-08 11:06

developer   ~51821

@ollehar : i already think of way to automatically reset cache in model : Question, Answer, Survey : afterSave function maybe. Maybe with conditio (code are updated, relevance are update : reset. Only help are updated : don't reset)

You can't remind when we move survey in SESSION (1.92 and 2.00 have a lot of issue about this) : all of issue‘s we have about deprecated survey in SESSION …

But in SESSION : it's only for current admin user, in cache : it's for ALL user …

ollehar

ollehar

2019-05-08 12:33

administrator   ~51827

Related commit (local method cache/memoization): https://github.com/LimeSurvey/LimeSurvey/commit/4c56d2025a4ed48d9f73972357df68ee65fbcd8f

DenisChenu

DenisChenu

2019-05-08 12:45

developer   ~51828

It's static var, not caching here …

For real aching

$value=Yii::app()->cache->get($id);
if(if($value===false) {
 $value = mycomplexsystem
 Yii::app()->cache->set($id,$value);
}
return $value;

I use it for a complex evaluation system, but create an alternative cache : one for cache DB used by LMS core, and one for my evaluation system. The hardest part is «when cache need to be resetted» but after it's easy.

Since it was a complete system with filters etc …

        private function getInCache($name,$aFilters=array())
        {
            if(isset(Yii::app()->session['user']) && Yii::app()->session['user']=='DenisChenu' && $this->debug)
            {
                return false;
            }
            $aBase=array();
            $aBase['limit']=$this->api->getRequest()->getParam('lastpage','1');
            $aFiltersBase=$this->api->getRequest()->getParam('filters');
            if(!empty($aFiltersBase))
                 $aBase = array_merge ( $aBase, $aFiltersBase );
            // number
            $numberFiltered=$this->aRenderData['countFiltered'];
            $cacheId="{$name}_{$this->iSurveyId}_{$numberFiltered}_".sha1(serialize(array_merge ($aBase, $aFilters )));
            return Yii::app()->cachembeem->get($cacheId);
        }

(You see my debug system too … :p )

ollehar

ollehar

2019-05-08 13:54

administrator   ~51830

Lots of good info about Yii caching in the docs: https://www.yiiframework.com/doc/guide/1.1/en/caching.fragment

ollehar

ollehar

2019-05-08 14:06

administrator   ~51831

Last edited: 2019-05-08 14:07

View 2 revisions

Example to cache a query:

$dependency = new CDbCacheDependency('SELECT MAX(update_time) FROM tbl_post');
$posts = Post::model()->cache(1000, $dependency)->with('comments')->findAll(array(
    'limit'=>20,
));
ollehar

ollehar

2019-05-08 14:22

administrator   ~51832

In case the meaning of the word "memoization" is unclear: https://en.wikipedia.org/wiki/Memoization

ollehar

ollehar

2019-05-08 14:52

administrator   ~51835

getLanguageData seem slow. Everything in gT() could probably be cached.

Selection_694.png (244,924 bytes)
ollehar

ollehar

2019-05-08 16:25

administrator   ~51836

Bah. gT is only slow when debug = 2 --> no cache enabled. With debug = 0, it's cached.

ollehar

ollehar

2019-05-08 16:43

administrator   ~51837

I could get req/sec up from 7 to 8 by caching getQuestionAttributesForEM in EM. 14% increase.

ollehar

ollehar

2019-05-08 17:07

administrator   ~51838

Pic shows that speed can improve 16% if setVariableAndTokenMappingsForExpressionManager is cached.

Selection_695.png (168,603 bytes)
DenisChenu

DenisChenu

2019-05-08 18:56

developer   ~51841

getLanguageData seem slow. Everything in gT() could probably be cached.

Yii already use caching for loadMessageSource : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/framework/i18n/CGettextMessageSource.php#L92

But use (by default) the cache system : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/application/config/internal.php#L200 set here : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/application/config/internal.php#L186

DenisChenu

DenisChenu

2019-05-08 18:57

developer   ~51842

It's a reason why i add SQLlite DB cache in my plugin : already available, just need runtimePath set out of web browser access.

DenisChenu

DenisChenu

2019-05-08 19:05

developer   ~51843

setVariableAndTokenMappingsForExpressionManager is a good starting point too, BUT : it currently use SESSION ? If i don't make error ?

ollehar

ollehar

2019-05-08 20:18

administrator   ~51844

Yes, I was working with debug = 2. >< So I didn't notice locale was already cached. Typical.

setVariableAndTokenMappingsForExpressionManager need to be refactored first, between side-effects and calculations. Not sure Carsten would like me to spend time on that for a potential 16% speed-up. :d

DenisChenu

DenisChenu

2019-05-09 07:55

developer   ~51847

potential 16% speed-up

Yes , sure :). It the most hard here : updating a big part can broke a lot and have a lot of bug after. Updating a lesser part didn't give you a lot …

DenisChenu

DenisChenu

2019-05-09 14:06

developer   ~51853

Question about different cache :
There are different cache system, some are great for a lot of little files loaded a load (memcache for example) some best for big data loaded not a load (but with a lot of SQL call and PHP script).
For example EM complete Session take sometimes more than 2 Mo, i think createFieldmap can take a lot of mem too : then maybe this need different cache system.

ollehar

ollehar

2019-05-10 14:14

administrator   ~51863

I think I agree with regard to create a separate cache object called emcache. This way, it can be turned off by default in the beginning while we test it, and later when always on, it can easily be turned off if there's a bug.

ollehar

ollehar

2019-05-10 17:50

administrator   ~51892

Some things will not be possible to cache if survey uses randomization.

DenisChenu

DenisChenu

2019-05-10 18:09

developer   ~51893

Last edited: 2019-05-10 18:11

View 4 revisions

For createFielsMap : can be cached here : https://github.com/LimeSurvey/LimeSurvey/blob/f62d45ef6c72bd16a65557b43e0c29070888314e/application/helpers/common_helper.php#L1947 no ?

For EM : it can be great if we can cache knowVar even partially , and merge it after with : TOKEN value
Need some related var, but we loose all this lines
If we can at same step : cache

We can not cache (and review if it's update knowVars )
gseq2info,

After get the new order : we merge update knowVars and q2subqInfo with the new gseq and qseq ?

I'm happy to make testing and give help on such project.

PS: it's awfull i put var name , and i have no idea of usage of this var …

ollehar

ollehar

2019-05-10 18:28

administrator   ~51894

Only bottlenecks should be cached. Use xdebug and kcachegrinde to analyze them. See my pics.

But I'll look more into details next week, if I get time. Thanks for discussion!

DenisChenu

DenisChenu

2019-05-10 18:36

developer   ~51895

For bottleneck : what is the survey tested ? It can be very different according to each survey , no ?

ollehar

ollehar

2019-05-10 18:42

administrator   ~51896

The bigger the survey, the more speed-up you will see. I used some I had lying around. The demo surveys included in the repo has a lot of EM logic, but number of questions affect a lot.

ollehar

ollehar

2019-05-23 12:51

administrator   ~52072

qanda might be possible to completely cache BEFORE any answers are posted/put in the session. How to check?

if (empty($_SESSION['survey_123'][$ia[1]])) {
  // Use cache
}
DenisChenu

DenisChenu

2019-05-23 12:52

developer   ~52074

Answers values can be get from default values that can came from previous data entered.

Unsure it's a good solution … to cache qanda

ollehar

ollehar

2019-05-23 13:13

administrator   ~52076

What does that mean, default values from previous data entered?

Caching qanda = high risk, but pretty high reward (not super high reward).

DenisChenu

DenisChenu

2019-05-23 15:35

developer   ~52088

default values from previous data entered

Yes, for example : qcode FirstName , default value {TOKEN:FIRSTNAME} at 1st page (1st group without welcome or all in one page) => all user after 1st can seen firstname of first user.

Then : «dynamic» part of qanda can not be cached.

Maybe there are other part : min max from {TOKEN:value} or value from URL ?

When adding this : need specific survey for testing :)

ollehar

ollehar

2019-05-23 16:21

administrator   ~52091

Damn. That's a good catch! Yes, qanda cache will be a little more complicated. We can still roll out emcache as a first step, and test qanda cache after.

ollehar

ollehar

2019-05-24 12:38

administrator   ~52102

@DenisChenu Can't find a manual page about caching in LS. You know of any?

DenisChenu

DenisChenu

2019-05-24 12:46

developer   ~52103

You mean caching in Yii ? I think we never create such page on manual of LimeSurvey …

ollehar

ollehar

2019-05-24 12:49

administrator   ~52104

OK, maybe I'll add something in the installation section.

DenisChenu

DenisChenu

2019-05-24 12:53

developer   ~52105

Maybe i add cache configuration in Installer update project : https://bugs.limesurvey.org/view.php?id=14772 :)

ollehar

ollehar

2019-05-24 23:11

administrator   ~52110

Last edited: 2019-05-24 23:18

View 3 revisions

Interesting. Looks like Sam tried to do something similar 5 years ago: https://github.com/LimeSurvey/LimeSurvey/commit/3bcc8b375ad2c217a1345bfcdacb40b25be47a25

It was reverted the same day. Wonder why. Maybe not possible if the $LEM variable is different for each user.

DenisChenu

DenisChenu

2019-05-25 12:11

developer   ~52111

I see you start checking caching qanda, but unsure we can (currently)
See this survey for example, where we use a question in next group (it's happen, really …)

If we cache qanda : such system must work (and i must hav too an dyn equation question type : if i remind we set it before qanda happen)

ollehar

ollehar

2019-05-25 18:24

administrator   ~52112

Thanks, will check on Monday. I just wanted to see what the possible speedup could be, but yes, it's only possible in a certain setup, so, limited use-case.

ollehar

ollehar

2019-05-27 15:30

administrator   ~52116

Maybe we need to consider adding an option to cache qanda. Seems hopelessly hard to detect it programmatically, like if any plugin subscribes to any event in retrieveAnswer, or if a certain expression is being used.

ollehar

ollehar

2019-05-28 13:18

administrator   ~52127

Changing language also does not work with qanda cache (the new version of the question does not get populated with the $_POST).

ollehar

ollehar

2019-05-28 16:59

administrator   ~52135

Merged with dev branch.

ollehar

ollehar

2019-12-18 12:32

administrator   ~55062

Bump. This could be worked on more if we ever get a R&D team.

Issue History

Date Modified Username Field Change
2019-05-07 20:10 ollehar New Issue
2019-05-07 20:10 ollehar Status new => assigned
2019-05-07 20:10 ollehar Assigned To => ollehar
2019-05-07 20:10 ollehar Note Added: 51816
2019-05-08 10:31 DenisChenu Note Added: 51817
2019-05-08 10:54 ollehar Note Added: 51818
2019-05-08 11:01 DenisChenu Note Added: 51819
2019-05-08 11:06 DenisChenu Note Added: 51821
2019-05-08 12:33 ollehar Note Added: 51827
2019-05-08 12:45 DenisChenu Note Added: 51828
2019-05-08 13:54 ollehar Note Added: 51830
2019-05-08 14:06 ollehar Note Added: 51831
2019-05-08 14:07 ollehar Note Edited: 51831 View Revisions
2019-05-08 14:22 ollehar Note Added: 51832
2019-05-08 14:52 ollehar File Added: Selection_694.png
2019-05-08 14:52 ollehar Note Added: 51835
2019-05-08 16:25 ollehar Note Added: 51836
2019-05-08 16:43 ollehar Note Added: 51837
2019-05-08 17:07 ollehar File Added: Selection_695.png
2019-05-08 17:07 ollehar Note Added: 51838
2019-05-08 18:56 DenisChenu Note Added: 51841
2019-05-08 18:57 DenisChenu Note Added: 51842
2019-05-08 19:05 DenisChenu Note Added: 51843
2019-05-08 20:18 ollehar Note Added: 51844
2019-05-09 07:55 DenisChenu Note Added: 51847
2019-05-09 14:06 DenisChenu Note Added: 51853
2019-05-10 14:14 ollehar Note Added: 51863
2019-05-10 17:50 ollehar Note Added: 51892
2019-05-10 18:09 DenisChenu Note Added: 51893
2019-05-10 18:09 DenisChenu Note Edited: 51893 View Revisions
2019-05-10 18:09 DenisChenu Note Edited: 51893 View Revisions
2019-05-10 18:11 DenisChenu Note Edited: 51893 View Revisions
2019-05-10 18:28 ollehar Note Added: 51894
2019-05-10 18:36 DenisChenu Note Added: 51895
2019-05-10 18:42 ollehar Note Added: 51896
2019-05-23 12:51 ollehar Note Added: 52072
2019-05-23 12:52 DenisChenu Note Added: 52074
2019-05-23 13:13 ollehar Note Added: 52076
2019-05-23 15:35 DenisChenu Note Added: 52088
2019-05-23 16:21 ollehar Note Added: 52091
2019-05-24 12:38 ollehar Note Added: 52102
2019-05-24 12:46 DenisChenu Note Added: 52103
2019-05-24 12:49 ollehar Note Added: 52104
2019-05-24 12:53 DenisChenu Note Added: 52105
2019-05-24 23:11 ollehar Note Added: 52110
2019-05-24 23:11 ollehar Note Edited: 52110 View Revisions
2019-05-24 23:18 ollehar Note Edited: 52110 View Revisions
2019-05-25 12:11 DenisChenu File Added: Capture d’écran du 2019-05-25 12-11-28.png
2019-05-25 12:11 DenisChenu File Added: limesurvey_survey_didQandaCanBeChached.lss
2019-05-25 12:11 DenisChenu Note Added: 52111
2019-05-25 18:24 ollehar Note Added: 52112
2019-05-27 15:30 ollehar Note Added: 52116
2019-05-27 16:23 ollehar Additional Information Updated View Revisions
2019-05-28 13:18 ollehar Note Added: 52127
2019-05-28 16:59 ollehar Note Added: 52135
2019-12-18 12:32 ollehar Note Added: 55062