Item8600: replace the inefficient pre-processed 'iterators' in Foswiki::Store::VC::Store with true iterators

in trunk, we've done away with the truely inefficient calls like @list = getTopicNames($web) (which force the core to pre-load the list of all topics into an array, taking time, and memory that we really should not do.

however, the job's only partially done. internally, the Store still implements the topic, web and attachments as a pre-processed array, which it then wraps into a ListIterator.

this means that in my 50,000 topic test web, every call to eachTopic on that web takes 500mS. and yes, we're not cacheing that list either.

I expect the same problem is true for companies that have a huge number of webs, and topics with a huge number of attachments - let alone lots of subwebs.

so... the change needed in trunk, is to convert all the Store methods that return an iterator, to actually do delayed processing iteration, and for bonus points, to separate the container from the iterator, such that the already iterated data can just be re-used elsewhere in the code, while un iterated datums are only evaluated as needed - with appropriate dealing with larger memory cases - there must be a compromise for massive cases, based on statistics on the reuse.

clearly, if the foswiki cache is enabled, more fun can ensue.

marking to me, but anyone that wants to implement it - have fun, and don't forget the unit tests.

-- SvenDowideit - 22 Feb 2010

Confirmed. If you sit in the "WaitingFor" field, no-one will even look, so changing it to "Anyone who likes a challenge".

BTW the sorting of topic names is delegated all the way down to the listing code.

-- CrawfordCurrie - 08 Apr 2010

Thinking further about this, and reading the code for all the tree iterator implementations on CPAN, I can't see that anyone has done it any better. I'd be happy to be corrected. Also, the VC store presents an iterator to the rest of the core, so I really don't think this is Urgent. Downgraded to enhancement.

-- CrawfordCurrie - 09 Apr 2010

Commenting is disabled while not logged in