You are here: Foswiki>Tasks Web>Item8600 (15 Mar 2013, GeorgeClark)Edit Attach

Item8600: replace the inefficient pre-processed 'iterators' in Foswiki::Store::VC::Store with true iterators

Priority: Enhancement
Current State: Confirmed
Released In: n/a
Target Release: minor
Applies To: Engine
Component: FoswikiStore, Performance
Reported By: SvenDowideit
Waiting For:
Last Change By: GeorgeClark
in trunk, we've done away with the truely inefficient calls like @list = getTopicNames($web) (which force the core to pre-load the list of all topics into an array, taking time, and memory that we really should not do.

however, the job's only partially done. internally, the Store still implements the topic, web and attachments as a pre-processed array, which it then wraps into a ListIterator.

this means that in my 50,000 topic test web, every call to eachTopic on that web takes 500mS. and yes, we're not cacheing that list either.

I expect the same problem is true for companies that have a huge number of webs, and topics with a huge number of attachments - let alone lots of subwebs.

so... the change needed in trunk, is to convert all the Store methods that return an iterator, to actually do delayed processing iteration, and for bonus points, to separate the container from the iterator, such that the already iterated data can just be re-used elsewhere in the code, while un iterated datums are only evaluated as needed - with appropriate dealing with larger memory cases - there must be a compromise for massive cases, based on statistics on the reuse.

clearly, if the foswiki cache is enabled, more fun can ensue.

marking to me, but anyone that wants to implement it - have fun, and don't forget the unit tests.

-- SvenDowideit - 22 Feb 2010

Confirmed. If you sit in the "WaitingFor" field, no-one will even look, so changing it to "Anyone who likes a challenge".

BTW the sorting of topic names is delegated all the way down to the listing code.

-- CrawfordCurrie - 08 Apr 2010

Thinking further about this, and reading the code for all the tree iterator implementations on CPAN, I can't see that anyone has done it any better. I'd be happy to be corrected. Also, the VC store presents an iterator to the rest of the core, so I really don't think this is Urgent. Downgraded to enhancement.

-- CrawfordCurrie - 09 Apr 2010


ItemTemplate edit

Summary replace the inefficient pre-processed 'iterators' in Foswiki::Store::VC::Store with true iterators
ReportedBy SvenDowideit
Codebase trunk
SVN Range
AppliesTo Engine
Component FoswikiStore, Performance
Priority Enhancement
CurrentState Confirmed
TargetRelease minor
ReleasedIn n/a
Topic revision: r6 - 15 Mar 2013, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy