Item12594: Some keyword searces run exponentially longer with more topics in web.

-- GeorgeClark - 10 Oct 2013

Reported on IRC: http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2013-10-09,Wed&sel=63#l59

A keyword search for "tmp" runs fine. A search for "/tmp" times out the server after 4 minutes or so. Memory grows rapidly during the apparent loop.

I've come up with a good recreation. It seems to be related to the number of files in a web.

dd if=/dev/urandom of=masterfile bs=1 count=1000000
base64 masterfile > masterout
find . -name "*.txt" -exec sed -i '1i %META:TOPICINFO{author="BaseUserMapping_333" comment="save topic" date="1381281879" format="1.1" reprev="2" version="2"}%' {} \;

That gave me around 1000 "topics".

Search for "Gp" found 308 hits and ran 1.7 seconds Search for "/Gp" returned 4 hits and ran 3.8 seconds.

I repeated the above generating another 1000 files from the same base data, so the number of topics doubled, should be 616 "Gp" topics and 8 containing "/Gp".

Search for "Gp" took 2.0 seconds Search for "/Gp" timed out after 4 minutes

---

Thanks to some help on IRC from pharvey who pointed me in the right direction. The issue is in foswiki/Meta.pm. The code that generates the $summary of the search hit is the root cause. Some print statements showed the search regex growing exponentially in length - every escape being escaped again for every hit.

Fix being checked in to trunk. Needs review.

-- GeorgeClark - 10 Oct 2013

Looks good

-- PaulHarvey - 10 Oct 2013

This also seems to happen when a period is in the search string, i.e. "hello.world", "what.time", "exim.conf", httpd.conf" -- Apache error logs, and dtrace show the same as before, and appears to still be related to the regex.

-- RyanSherer - 20 Feb 2014

Once a task has been closed as fixed and shipped in a release, we can't do any more work against it. Could you please open a new task to track this issue.

We are also going to need a bit more trouibleshooting information. Are you running 1.1.9, or a patched 1.1.8? What are your search settings in configure, etc. As I commented in some email discussion, I wasn''t able to recreate the new issue on either a local test system or on foswiki.org. I just tried it again from this page and it took 18 seconds to search for exim.conf, but it definitely didn't loop, and returned 1 hit (this topic).

-- GeorgeClark - 20 Feb 2014

Commenting is disabled while not logged in