@DenisChenu Thoughts?

I think i already put a Dev part about this. And it's a really good idea.

The point :

1. We must clean/disable cache when admin user update survey : then create a new function
2. Current createFielsMap get : url params + random + token value : must separate
3. We must do same things for some EM function https://github.com/LimeSurvey/LimeSurvey/blob/ffee4e19f81f9d0130fb033d394d6d8540ba0fcc/application/helpers/expressions/em_manager_helper.php#L4987 maybe ?

1. Yep, saving anything will mark the cache as dirty. The cache is only active when the survey is activated.
2. True, some parts are individual and need to be overwritten.
3. I did a benchmark of LS - EM run preg_match 180k times during one survey load! (Demo survey.)

1. Not only : current EM session contains question text : i think we must remove it from session.

Maybe we must remove some part for

1. Session (to be moved to cache only)
2. Cache (to be moved to DB only)
3. Session only (Token value, current answers value, prefill url value …), maybe separate session : survey_XXXX and livesurvey_XXX

For example : i‘m unsure we need question.text : i think it's to have it in js expression : but it false : https://bugs.limesurvey.org/view.php?id=14817 , then we can, call it in EM from DB when needed (One DB line call : quick).

PS : i really think it's a REALY complex system and come with a lot of issues …

@ollehar : i already think of way to automatically reset cache in model : Question, Answer, Survey : afterSave function maybe. Maybe with conditio (code are updated, relevance are update : reset. Only help are updated : don't reset)

You can't remind when we move survey in SESSION (1.92 and 2.00 have a lot of issue about this) : all of issue‘s we have about deprecated survey in SESSION …

But in SESSION : it's only for current admin user, in cache : it's for ALL user …

Related commit (local method cache/memoization): https://github.com/LimeSurvey/LimeSurvey/commit/4c56d2025a4ed48d9f73972357df68ee65fbcd8f

It's static var, not caching here …

For real aching

$value=Yii::app()->cache->get($id);
if(if($value===false) {
 $value = mycomplexsystem
 Yii::app()->cache->set($id,$value);
}
return $value;

I use it for a complex evaluation system, but create an alternative cache : one for cache DB used by LMS core, and one for my evaluation system. The hardest part is «when cache need to be resetted» but after it's easy.

Since it was a complete system with filters etc …

        private function getInCache($name,$aFilters=array())
        {
            if(isset(Yii::app()->session['user']) && Yii::app()->session['user']=='DenisChenu' && $this->debug)
            {
                return false;
            }
            $aBase=array();
            $aBase['limit']=$this->api->getRequest()->getParam('lastpage','1');
            $aFiltersBase=$this->api->getRequest()->getParam('filters');
            if(!empty($aFiltersBase))
                 $aBase = array_merge ( $aBase, $aFiltersBase );
            // number
            $numberFiltered=$this->aRenderData['countFiltered'];
            $cacheId="{$name}_{$this->iSurveyId}_{$numberFiltered}_".sha1(serialize(array_merge ($aBase, $aFilters )));
            return Yii::app()->cachembeem->get($cacheId);
        }

(You see my debug system too … :p )

Lots of good info about Yii caching in the docs: https://www.yiiframework.com/doc/guide/1.1/en/caching.fragment

Example to cache a query:

$dependency = new CDbCacheDependency('SELECT MAX(update_time) FROM tbl_post');
$posts = Post::model()->cache(1000, $dependency)->with('comments')->findAll(array(
    'limit'=>20,
));

In case the meaning of the word "memoization" is unclear: https://en.wikipedia.org/wiki/Memoization

getLanguageData seem slow. Everything in gT() could probably be cached.

Bah. gT is only slow when debug = 2 --> no cache enabled. With debug = 0, it's cached.

I could get req/sec up from 7 to 8 by caching getQuestionAttributesForEM in EM. 14% increase.

Pic shows that speed can improve 16% if setVariableAndTokenMappingsForExpressionManager is cached.

getLanguageData seem slow. Everything in gT() could probably be cached.

Yii already use caching for loadMessageSource : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/framework/i18n/CGettextMessageSource.php#L92

But use (by default) the cache system : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/application/config/internal.php#L200 set here : https://github.com/LimeSurvey/LimeSurvey/blob/70d03f7ba093700c28f3727cd04760558dea17bb/application/config/internal.php#L186

It's a reason why i add SQLlite DB cache in my plugin : already available, just need runtimePath set out of web browser access.

setVariableAndTokenMappingsForExpressionManager is a good starting point too, BUT : it currently use SESSION ? If i don't make error ?

Yes, I was working with debug = 2. >< So I didn't notice locale was already cached. Typical.

setVariableAndTokenMappingsForExpressionManager need to be refactored first, between side-effects and calculations. Not sure Carsten would like me to spend time on that for a potential 16% speed-up. :d

potential 16% speed-up

Yes , sure :). It the most hard here : updating a big part can broke a lot and have a lot of bug after. Updating a lesser part didn't give you a lot …

Question about different cache :
There are different cache system, some are great for a lot of little files loaded a load (memcache for example) some best for big data loaded not a load (but with a lot of SQL call and PHP script).
For example EM complete Session take sometimes more than 2 Mo, i think createFieldmap can take a lot of mem too : then maybe this need different cache system.

I think I agree with regard to create a separate cache object called emcache. This way, it can be turned off by default in the beginning while we test it, and later when always on, it can easily be turned off if there's a bug.

Some things will not be possible to cache if survey uses randomization.

For createFielsMap : can be cached here : https://github.com/LimeSurvey/LimeSurvey/blob/f62d45ef6c72bd16a65557b43e0c29070888314e/application/helpers/common_helper.php#L1947 no ?

For EM : it can be great if we can cache knowVar even partially , and merge it after with : TOKEN value
Need some related var, but we loose all this lines
If we can at same step : cache

• qattr (this one ca be cached easily now ?).
• qans
• q2subqInfo
• qcode2sgqa
• jsVar2qid
• qcode2sgq

We can not cache (and review if it's update knowVars )
gseq2info,

After get the new order : we merge update knowVars and q2subqInfo with the new gseq and qseq ?

I'm happy to make testing and give help on such project.

PS: it's awfull i put var name , and i have no idea of usage of this var …

Only bottlenecks should be cached. Use xdebug and kcachegrinde to analyze them. See my pics.

But I'll look more into details next week, if I get time. Thanks for discussion!

For bottleneck : what is the survey tested ? It can be very different according to each survey , no ?

The bigger the survey, the more speed-up you will see. I used some I had lying around. The demo surveys included in the repo has a lot of EM logic, but number of questions affect a lot.

qanda might be possible to completely cache BEFORE any answers are posted/put in the session. How to check?

if (empty($_SESSION['survey_123'][$ia[1]])) {
  // Use cache
}

Answers values can be get from default values that can came from previous data entered.

Unsure it's a good solution … to cache qanda

What does that mean, default values from previous data entered?

Caching qanda = high risk, but pretty high reward (not super high reward).

default values from previous data entered

Yes, for example : qcode FirstName , default value {TOKEN:FIRSTNAME} at 1st page (1st group without welcome or all in one page) => all user after 1st can seen firstname of first user.

Then : «dynamic» part of qanda can not be cached.

Maybe there are other part : min max from {TOKEN:value} or value from URL ?

When adding this : need specific survey for testing :)

Damn. That's a good catch! Yes, qanda cache will be a little more complicated. We can still roll out emcache as a first step, and test qanda cache after.

@DenisChenu Can't find a manual page about caching in LS. You know of any?

You mean caching in Yii ? I think we never create such page on manual of LimeSurvey …

OK, maybe I'll add something in the installation section.

Maybe i add cache configuration in Installer update project : https://bugs.limesurvey.org/view.php?id=14772 :)

Interesting. Looks like Sam tried to do something similar 5 years ago: https://github.com/LimeSurvey/LimeSurvey/commit/3bcc8b375ad2c217a1345bfcdacb40b25be47a25

It was reverted the same day. Wonder why. Maybe not possible if the $LEM variable is different for each user.

I see you start checking caching qanda, but unsure we can (currently)
See this survey for example, where we use a question in next group (it's happen, really …)

If we cache qanda : such system must work (and i must hav too an dyn equation question type : if i remind we set it before qanda happen)

Thanks, will check on Monday. I just wanted to see what the possible speedup could be, but yes, it's only possible in a certain setup, so, limited use-case.

Maybe we need to consider adding an option to cache qanda. Seems hopelessly hard to detect it programmatically, like if any plugin subscribes to any event in retrieveAnswer, or if a certain expression is being used.

Changing language also does not work with qanda cache (the new version of the question does not get populated with the $_POST).

Merged with dev branch.

Bump. This could be worked on more if we ever get a R&D team.