Item9015: Using ATTACHMENTLIST in a query search causes the search to return only SOME of the results
Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a
I have a web with about 3000 topics each of which has a 10 field form attached. If I use the following query:
%SEARCH{
type="query"
search="DocumentType='Document' AND DocumentStatus='Transferred'"
format="$percntATTACHMENTLIST{topic=$quot$topic$quot web=$quot$web$quot header=$quot * [[$web.$topic][$topic]] has $fileCount files(s)$quot format=$quot * $fileName$quot }$percnt"
nosearch="on"
}%
The search says:
Number of topics: 1267
But
only about 80 show up in output. A whole bunch are missing.
Apache throws a slew of errors like this:
[Wed May 12 19:23:18 2010] [error] [client 192.168.254.192] [Wed May 12 19:23:18 2010] view: readdir() attempted on invalid dirhan\
dle $DIR at /var/lib/foswiki/lib/Foswiki/Store/RcsFile.pm line 256., referer: (redacted)/foswiki/bin/edit/(redacted)?t=1273717053;nowysiwyg=1
[Wed May 12 19:23:18 2010] [error] [client 192.168.254.192] [Wed May 12 19:23:18 2010] view: closedir() attempted on invalid dirha\
ndle $DIR at /var/lib/foswiki/lib/Foswiki/Store/RcsFile.pm line 265., referer: (redacted)/foswiki/bin/edit/(redacted)?t=1273717053;nowysiwyg=1
If I change the search to specifically look for one of the missing topics, like this:
search="name='thetopicname'"
The search says "Number of topics: 1", but
the topic isn't listed!
If I edit and save the topic and repeat this (single-topic) query, the topic
does get listed.
I'm using
AttachmentListPlugin (1.3.3, $Rev: 4282 (2009-06-22) $)
--
BigBaaadBob - 13 May 2010
I also get this error on the same lines in
RcsFile.pm
, but using
DBCachePlugin. Problem seems to be DBQUERY trying to build a cache for a web that doesn't exist; but the problem is
really hard for me to reproduce.
I discovered it was trying to open a non-existent web called "WikiTopics" or something similar, and assume my custom
DBCacheContrib code (or usage of DBQUERY macro) is doing something bad.
I discovered the name of this phantom web using code similar to the following
--- /usr/local/src/Foswiki-1.0.9/lib/Foswiki/Store/RcsFile.pm 2010-01-18 01:16:20.000000000 +1100
+++ RcsFile.pm 2010-05-05 13:32:03.000000000 +1000
@@ -250,9 +250,9 @@
sub getTopicNames {
my $this = shift;
- opendir my $DIR, $Foswiki::cfg{DataDir} . '/' . $this->{web};
-
my @topicList;
+ if (opendir my $DIR, $Foswiki::cfg{DataDir} . '/' . $this->{web}) {
+ print STDERR 'Got dirhandle for web: ' . $this->{web};
foreach my $f (readdir($DIR)) {
# Validate and untaint
next unless $f =~ /^(.*)\.txt$/;
@@ -264,6 +264,11 @@
}
closedir($DIR);
return sort @topicList;
+ } else {
+ print STDERR 'Didn\'t get valid dirhandle for web: ' . $this->{web};
+ }
+
+ return;
}
It's interesting to note that
BigBaaadBob and I are both running Perl 5.10; or maybe that's just coincidence. I have noticed instabilities in
DBCacheContrib since upgrading our server; spurious uninitialised variable warnings, and so on....
--
PaulHarvey
It turns out that the Apache errors were caused by the
DBCacheContrib. I removed it, its plugin, and
FormQueryPlugin from the system and the errors went away. See:
http://foswiki.org/Tasks/Item9016.
Unfortunately, the above behaviour (missing topic) still remains. Further debugging shows that this happens when topics have auto-attached files. IOW, if a topic has auto-attached files, the query above will not return them in the listing even though the query knows about them.
The easiest way to reproduce is this:
- Create a topic with an attached file.
- Modify and use the above query to verify that it works on that topic.
- Be sure you have Auto-attach enabled.
- Edit the topic text on the server in an editor and delete the attachment meta-text.
- Verify the attachment still shows up in Fiswiki (auto-attach is working).
- Verify the above query does not list the topic.
I upped this to Urgent because ATTACHMENTLIST completely fails to work with auto-attachments.
--
BigBaaadBob - 13 May 2010
the ATTACHMENTLIST macro reads the list of attachments from the topic Metadata. The Func::getAttachmentList appears to internally iterate the attachment directory instead of just using Meta, but unfortunately in 1.0.9, the Func::getAttachmentList API does not yet exist.
Plugin needs to be changed to iterate the directory on disk and recover the attachment names from the actual files. eventually for 1.1, it could probably use the Func::getAttachmentlist to populate the list.
--
GeorgeClark - 13 May 2010
I've committed a fix and unit test to SVN,
--
GeorgeClark - 14 May 2010
Tested this and it seems to work fine, without any major performance issues.
--
BigBaaadBob - 18 May 2010
Great! Thanks for this improvement.
--
ArthurClemens - 18 May 2010