Item10720: SolrIndex fails on Form Definition

I think this happens when the attached form is missing (but haven't confirmed).

| 2011-05-07T15:26:10Z warning | Can't use an undefined value as an ARRAY reference at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SolrPlugin/Index.pm line 332.
 at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SolrPlugin/Index.pm line 332
        Foswiki::Plugins::SolrPlugin::Index::indexTopic('Foswiki::Plugins::SolrPlugin::Index=HASH(0x4302048)', 'Mangroves', 'HasDetritivore', 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)', '%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SolrPlugin/Index.pm line 211
        Foswiki::Plugins::SolrPlugin::Index::updateTopic('Foswiki::Plugins::SolrPlugin::Index=HASH(0x4302048)', 'Mangroves', 'HasDetritivore', 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)', '%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SolrPlugin/Index.pm line 95
        Foswiki::Plugins::SolrPlugin::Index::afterSaveHandler('Foswiki::Plugins::SolrPlugin::Index=HASH(0x4302048)', 'Mangroves', 'HasDetritivore', 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)', '%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SolrPlugin.pm line 184
        Foswiki::Plugins::SolrPlugin::afterSaveHandler('%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...', 'HasDetritivore', 'Mangroves', undef, 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugin.pm line 285
        Foswiki::Plugin::invoke('Foswiki::Plugin=HASH(0x3e5bb80)', 'afterSaveHandler', '%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...', 'HasDetritivore', 'Mangroves', undef, 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins.pm line 331
        Foswiki::Plugins::dispatch('Foswiki::Plugins=HASH(0x25d3e10)', 'afterSaveHandler', '%META:TOPICINFO{author="BaseUserMapping_333" date="1304781969...', 'HasDetritivore', 'Mangroves', undef, 'Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Meta.pm line 2004
        Foswiki::Meta::save('Foswiki::Plugins::MongoDBPlugin::Meta=HASH(0x5c2c278)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SemanticLinksPlugin/Core.pm line 545
        Foswiki::Plugins::SemanticLinksPlugin::Core::restReparseHandler('Foswiki=HASH(0x25d32e8)', 'SemanticLinksPlugin', 'reparse', 'Foswiki::Response=HASH(0x25c7a18)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Plugins/SemanticLinksPlugin.pm line 110
        Foswiki::Plugins::SemanticLinksPlugin::restReparseHandler('Foswiki=HASH(0x25d32e8)', 'SemanticLinksPlugin', 'reparse', 'Foswiki::Response=HASH(0x25c7a18)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Func.pm line 665
        Foswiki::Func::__ANON__('Foswiki=HASH(0x25d32e8)', 'SemanticLinksPlugin', 'reparse', 'Foswiki::Response=HASH(0x25c7a18)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/UI/Rest.pm line 249
        Foswiki::UI::Rest::rest('Foswiki=HASH(0x25d32e8)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/UI.pm line 316
        Foswiki::UI::__ANON__() called at /usr/local/share/perl/5.10.1/Error.pm line 415
        eval {...} called at /usr/local/share/perl/5.10.1/Error.pm line 407
        Error::subs::try('CODE(0x1ec8d40)', 'HASH(0x25daf80)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/UI.pm line 435
        Foswiki::UI::_execute('Foswiki::Request=HASH(0x25c8108)', 'CODE(0x25c7d60)', 'rest', 1, 'command_line', 1) called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/UI.pm line 274
        Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x25c8108)') called at /usr/local/src/git.trin.org.au/core/lib/Foswiki/Engine/CLI.pm line 53
        Foswiki::Engine::CLI::run('Foswiki::Engine::CLI=HASH(0x1f94340)') called at bin/./rest line 29.
 |

Here's our patch (just skipping the loop if getFields() returns undef)

diff --git a/lib/Foswiki/Plugins/SolrPlugin/Index.pm b/lib/Foswiki/Plugins/SolrPlugin/Index.pm
index 618863a..87c751f 100644
--- a/lib/Foswiki/Plugins/SolrPlugin/Index.pm
+++ b/lib/Foswiki/Plugins/SolrPlugin/Index.pm
@@ -329,56 +329,59 @@ sub indexTopic {
     if ($formDef) { # form definition found, if not the formfields aren't indexed
 
       my %seenFields = ();
-      foreach my $fieldDef (@{$formDef->getFields()}) {
-        my $attrs = $fieldDef->{attributes}; # TODO: check for Facet
-        my $name = $fieldDef->{name};
-        my $type = $fieldDef->{type};
-        my $field = $meta->get('FIELD', $name);
-        next unless $field;
-
-        # prevent from mall-formed formDefinitions 
-        if ($seenFields{$name}) {
-          $this->log("WARNING: walrofmed form definition for $web.$formName - field $name appear twice must be unique");
-          next;
-        }
-        $seenFields{$name} = 1;
-
-        my $value = $field->{value};
-
-        # create a dynamic field indicating the field type to solr
-
-        # date
-        if ($type eq 'date') {
-          my $epoch = $value;
-          $epoch = Foswiki::Time::parseTime($value) unless $epoch =~ /^\d+$/;
-          $epoch ||= 0; # prevent formatTime to crap out
-          $value = Foswiki::Time::formatTime($epoch, 'iso', 'gmtime');
-          $doc->add_fields(
-            'field_'.$name.'_dt' => $value,
-          );
-        } 
-
-        # multi-valued types
-        elsif ($type =~ /^(checkbox|select|radio|textboxlist)$/ ||
-               $name =~ /TopicType/) { # TODO: make this configurable
-          foreach my $item (split(/\s*,\s*/, $value)) {
-            $doc->add_fields(
-              'field_'.$name.'_lst' => $item
-            );
-          }
-        }
-
-        # make it a text field unless its name does not indicate otherwise
-        else {
-          my $fieldName = 'field_'.$name;
-          my $fieldType = '_s';
-          if ($fieldName =~ /(_(?:i|s|l|t|b|f|dt|lst))$/) {
-            $fieldType = $1;
-          }
-          $doc->add_fields(
-            $fieldName.$fieldType => $value,
-          );
-        }
+      my $formFields = $formDef->getFields();
+      if ($formFields) {
+            foreach my $fieldDef (@{$formFields}) {
+               my $attrs = $fieldDef->{attributes}; # TODO: check for Facet
+               my $name = $fieldDef->{name};
+               my $type = $fieldDef->{type};
+               my $field = $meta->get('FIELD', $name);
+               next unless $field;
+
+               # prevent from mall-formed formDefinitions 
+               if ($seenFields{$name}) {
+                  $this->log("WARNING: walrofmed form definition for $web.$formName - field $name appear twice must be unique");
+                  next;
+               }
+               $seenFields{$name} = 1;
+
+               my $value = $field->{value};
+
+               # create a dynamic field indicating the field type to solr
+
+               # date
+               if ($type eq 'date') {
+                  my $epoch = $value;
+                  $epoch = Foswiki::Time::parseTime($value) unless $epoch =~ /^\d+$/;
+                  $epoch ||= 0; # prevent formatTime to crap out
+                  $value = Foswiki::Time::formatTime($epoch, 'iso', 'gmtime');
+                  $doc->add_fields(
+                     'field_'.$name.'_dt' => $value,
+                  );
+               } 
+
+               # multi-valued types
+               elsif ($type =~ /^(checkbox|select|radio|textboxlist)$/ ||
+                         $name =~ /TopicType/) { # TODO: make this configurable
+                  foreach my $item (split(/\s*,\s*/, $value)) {
+                     $doc->add_fields(
+                        'field_'.$name.'_lst' => $item
+                     );
+                  }
+               }
+
+               # make it a text field unless its name does not indicate otherwise
+               else {
+                  my $fieldName = 'field_'.$name;
+                  my $fieldType = '_s';
+                  if ($fieldName =~ /(_(?:i|s|l|t|b|f|dt|lst))$/) {
+                     $fieldType = $1;
+                  }
+                  $doc->add_fields(
+                     $fieldName.$fieldType => $value,
+                  );
+               }
+            }
       }
     }
   }

-- PaulHarvey - 07 May 2011

As far as I can see most is whitespace changes. Not sure. Paul, could you please provide a better patch with only the bug fix in it and no whitespace changes at all. Thanks.

-- MichaelDaum - 08 May 2011

Sorry for the delay. Here's a version without whitespace changes

diff --git a/lib/Foswiki/Plugins/SolrPlugin/Index.pm b/lib/Foswiki/Plugins/SolrP
index 931f6cc..381a87f 100644
--- a/lib/Foswiki/Plugins/SolrPlugin/Index.pm
+++ b/lib/Foswiki/Plugins/SolrPlugin/Index.pm
@@ -329,7 +329,9 @@ sub indexTopic {
     if ($formDef) { # form definition found, if not the formfields aren't index
 
       my %seenFields = ();
-      foreach my $fieldDef (@{$formDef->getFields()}) {
+      my $formFields = $formDef->getFields();
+      if ($formFields) {
+      foreach my $fieldDef (@{$formFields}) {
         my $attrs = $fieldDef->{attributes}; # TODO: check for Facet
         my $name = $fieldDef->{name};
         my $type = $fieldDef->{type};
@@ -380,6 +382,7 @@ sub indexTopic {
           );
         }
       }
+     }
     }
   }

-- PaulHarvey - 30 May 2011

OIC.

-- MichaelDaum - 30 May 2011

We have still have this issue with the latest Solr-Plugin. In our case, the Form IS actually there. Just checked Index.pm. Looks like the above Patch is there.

The Problem is, after getting the error

ERROR: can't read form definition for Applications.Tasks.TaskData
Error: Can't use an undefined value as an ARRAY reference

Indexing stops completely. So, we never can do a full Index. Interestingly, it is ONLY happening in ONE certain web and there only on certain topics (all have the same form attached). We use forms in uncountable other webs and topics as well, and there we don't have this problem. If we copy those topics to another web, the error happens there too, though.

Here, the Meta-Information from one of those Topics:

% META:TOPICINFO{author="lichtsteiner" date="1339157772" format="1.1" version="3"}%

Topic Content.....

% META:FORM{name="Applications.Tasks.TaskData"}%
% META:FIELD{name="TopicTitle" attributes="H" title="<nop>TopicTitle" value="Features freigegeben im April 2012"}%
% META:FIELD{name="ReportedBy" attributes="" title="<nop>ReportedBy" value="Main.OliverSchaub"}%
% META:FIELD{name="AppliesTo" attributes="" title="<nop>AppliesTo" value="varia"}%
% META:FIELD{name="Due" attributes="" title="Due" value=""}%
% META:FIELD{name="Priority" attributes="" title="Priority" value=""}%
% META:FIELD{name="Type" attributes="" title="Type" value="task"}%
% META:FIELD{name="CurrentState" attributes="" title="<nop>CurrentState" value="Closed"}%
% META:FIELD{name="WaitingFor" attributes="M" title="<nop>WaitingFor" value="Main.OliverSchaub"}%

Here, the Form "TaskData" under Applications.Tasks:

% META:TOPICINFO{author="schaub_5fo" comment="save topic" date="1377870883" format="1.1" reprev="2" version="7"}%
---++ This is the %SYSTEMWEB%.DataForms definition for the structured form based data
see WebTopicEditTemplate for the TopicTemplate

<verbatim>
| *Name*            | *Type* | *Size* | *Values* | *Tooltip message* | *Attributes* |
| <nop>TopicTitle   | label | 1 |  |  | H |
| <nop>ReportedBy   | text   | 35     | %WIKIUSERNAME% | | |
| <nop>AppliesTo | select | 1 | %FORMATLIST{"%SEARCH{"^AppliesTo.+" scope="topic" type="regex" web="TestWeb" excludetopic="AppliesToActiveDirectoryErneuerung,AppliesToAITWeb,AppliesToiDokMigration" format="$percentTOPICTITLE{\"$topic\"}$percent" nonoise="on" separator=", "}%%IF{"'%BASETOPIC%'/AppliesTo" then=", %QUERY{"'%BASETOPIC%'/AppliesTo"}%"}%" split="," separator="," unique="on"}% | | |
| Due               | text   | 30 |  | | |
| Priority          | select |  | , low, high | | |
| Type              | select |  | task, bug, change request | | |
| <nop>CurrentState | select |  | Discussion, Confirmed, Being Worked On, Closed |  |  |
| <nop>WaitingFor   | text   | 35 | | | M |
</verbatim>

Viewing these Topics works without problems, the forms are displayed correctly.

If only the indexer would ignore this and continue indexing. This would help us a lot as a workaround.

-- OliverSchaub - 05 Sep 2013

I added some whitespaces in the examples above. (% META)

-- OliverSchaub - 05 Sep 2013

Did you try when removing the <verbatim> tag in the form definition? The form won't display on a normal view either that way. That's very uncommon.

-- MichaelDaum - 05 Sep 2013

I guess the verbatim was there so the Search was not executed while watching the Form-Topic.

I removed the Verbatim-Tag. The Error is still the same.

-- OliverSchaub - 05 Sep 2013

Can't reproduce.

What I get is:

Indexing topic Sandbox.TaskTest1
ERROR: can't read form definition for TaskForm
ERROR: can't read form definition TaskForm in ClassificationPlugin::Core::getIndexFields
Committing index

But that's inside my dev environment...not representative.

A stack trace for this error would help to nail down your issue.

-- MichaelDaum - 05 Sep 2013

I attached the trace of a single topic-index.

-- OliverSchaub - 05 Sep 2013

Argh, an strace. I was more referring to a print-out of perl's call stack.

-- MichaelDaum - 05 Sep 2013

OK. This somehow exceeds my Perl-Knowledge... I'll have to check how I can get this Perl stack-trace.

-- OliverSchaub - 09 Sep 2013

OK. Looks like we've solved the Problem... I tried to create this Perl Stack-Trace with "MCarp". For this, I updated the existing Carp-Modules. After Updating "perl-Devel-Carp", the Indexer ran through without any Problems.

So, obviously, a "broken" Perl-Module caused the indexer to fail on certain Topics.

Thanks for your Help, I will close the Task for now.

-- OliverSchaub - 16 Sep 2013

Commenting is disabled while not logged in