You are here: Foswiki>Tasks Web>Item14067 (13 May 2017, GeorgeClark)Edit Attach

Item14067: Reimplement Foswiki::MetaCache

pencil
Priority: Enhancement
Current State: Confirmed
Released In: 2.2.0
Target Release: minor
Applies To: Engine
Component:
Branches:
Reported By: MichaelDaum
Waiting For:
Last Change By: GeorgeClark
... and rethink caching of meta objects.

The Foswiki::MetaCache class is basically unused in that nothing takes advantage of it. Yet still other code paths pump entries into it. For nothing. except the Foswiki::Search::* code. Nothing else profits from it.

The basic idea of caching Foswiki::Meta objects is a good one and Devel::NYTProf shows that we spent most of the time re-re-re-reading the same topic revision multiple times. The bulk work loading a topic is converting plain text %META:... strings to structured perl data via _readKeyValues. This happens a lot. So caching a meta object seems to be the RightThingToDo_(tm).

Alas, the current implementation of a Foswiki::MetaCache is hanging off the Foswiki::Search code where it clearly doesn't belong to. The Foswiki::MetaCache should be a service of the Foswiki session/app, that is then used by Foswiki::Meta::load().

The current impl tries to cache per user which is not required as after finishing a request will the Foswiki::MetaCache be destroyed anyway. Even when it persistet multiple requests are the topics the same on disc independent of any user looking at them ... granted a user is not modifying a topic object as part of the request, of course. If that was the case will we need to clone a topic object before to create a private copy not stored in the Foswiki::MetaCache.

For now - and part of a next patch release - all references to the Foswiki::MetaCache should be removed from the code as a first step. It is unused and costing performance for nothing.

This is not correct! The MetaCache is indeed used, and is critical for good Query Search performance. This fix distro:4e7a76058509 resulted in a 350% improvement in search performance. This change is too big for a patch release. Please do not make these changes in 2.1.3. -- GeorgeClark

In a second - and only part of a next minor release - step Foswiki::MetaCache needs a radical simplification.

And then integrate it back again at the right position of the code path again, either as part of the Foswiki::Store api or one level above in Foswiki::Meta (doesn't matter much).
The role of the MetaCache is not to just avoid I/O, but to also cache the de-serialization that happens when the stored topic is instantiated in a Meta object. I think a cache of the Meta object is quite critical, and caching the Stored object won't help much. -- GeorgeClark

We will then benchmark whether that pays off for a single request already, which I assume as poor-mans debugging readTopic() shows loads of redundant operations here for one single request.

Finally, we implement some kind of Least-Recently-Used exclusion strategy and try keeping a Foswiki::MetaCache hanging around for multiple requests (given a persistent perl env).

Btw, here is a treemap of a %SEARCH over a web with >2k topics. I've marked all code paths that are involved in creating topic objects.

FoswikiProfileSearch1.png

-- MichaelDaum - 10 May 2016

Caching was discussed few times as a part of OO design. Jozef was mentioning CPAN:CHI module and it was generally agreed that it's a good way to go as it allows caching of any kind of objects and robust.

-- VadimBelman - 10 Aug 2016

Changed this to an enhancement.

-- GeorgeClark - 13 May 2017
 

ItemTemplate edit

Summary Reimplement Foswiki::MetaCache
ReportedBy MichaelDaum
Codebase 2.1.2, trunk
SVN Range
AppliesTo Engine
Component
Priority Enhancement
CurrentState Confirmed
WaitingFor
Checkins
TargetRelease minor
ReleasedIn 2.2.0
CheckinsOnBranches
trunkCheckins
masterCheckins
ItemBranchCheckins
Release02x01Checkins
Release02x00Checkins
Release01x01Checkins
I Attachment Action Size Date Who Comment
FoswikiProfileSearch1.pngpng FoswikiProfileSearch1.png manage 126 K 10 May 2016 - 10:10 MichaelDaum  
Topic revision: r6 - 13 May 2017, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy