Item13181: Config.spec PERL type scrambled, and configure crashes if PERL type contains any embedded comments.

Priority: Urgent
Current State: Closed
Released In: 2.0.0
Target Release: major

Applies To: Engine
Component: Configure, ConfigurePlugin
Branches: master

Reported By: GeorgeClark
Waiting For:
Last Change By: GeorgeClark

$Foswiki::cfg{Plugins}{PerlPlugin}{Share} = {
    'Foswiki::Func' => [
        '&getSkin',
        #'&getUrlHost',
        '&getScriptUrl',
       ]
    };

Foswiki configure
Error
Foswiki::Configure::Query::getspec errors: /var/www/foswiki/distro/core/lib/Foswiki/Plugins/PerlPlugin/Config.spec: 19: Cannot eval value '{ 'Foswiki::Func' => [ '&getSkin', #'&getUrlHost', '&getScriptUrl', ] }': syntax error at (eval 897) line 1, at EOF
Missing right curly or square bracket at (eval 897) line 1, at end of line
at /usr/lib/perl5/vendor_perl/5.18.2/CGI/Carp.pm line 378.
CGI::Carp::realdie('Foswiki::Configure::Query::getspec errors: /var/www/foswiki/d...') called at /usr/lib/perl5/vendor_perl/5.18.2/CGI/Carp.pm line 467
CGI::Carp::die('Foswiki::Configure::Query::getspec errors: /var/www/foswiki/d...') called at /var/www/foswiki/distro/core/lib/Foswiki/Plugins/ConfigurePlugin.pm line 171
Foswiki::Plugins::ConfigurePlugin::__ANON__('Foswiki=HASH(0x883bbf0)', 'Foswiki::Contrib::JsonRpcContrib::Request=HASH(0x8dedf00)', 'Foswiki::Response=HASH(0x883bcc0)', undef) called at /var/www/foswiki/distro/core/lib/Foswiki/Contrib/JsonRpcContrib/Server.pm line 175
Foswiki::Contrib::JsonRpcContrib::Server::__ANON__() called at /usr/lib/perl5/vendor_perl/5.18.2/Error.pm line 419
eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/Error.pm line 411
Error::subs::try('CODE(0x8fd3408)', 'HASH(0x8fd35b8)') called at /var/www/foswiki/distro/core/lib/Foswiki/Contrib/JsonRpcContrib/Server.pm line 189
Foswiki::Contrib::JsonRpcContrib::Server::dispatch('Foswiki::Contrib::JsonRpcContrib::Server=HASH(0x8e0e710)', 'Foswiki=HASH(0x883bbf0)') called at /var/www/foswiki/distro/core/lib/Foswiki/Contrib/JsonRpcContrib.pm line 48
Foswiki::Contrib::JsonRpcContrib::dispatch('Foswiki=HASH(0x883bbf0)') called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 372
Foswiki::UI::__ANON__() called at /usr/lib/perl5/vendor_perl/5.18.2/Error.pm line 419
eval {...} called at /usr/lib/perl5/vendor_perl/5.18.2/Error.pm line 411
Error::subs::try('CODE(0x804dd38)', 'HASH(0x88133b8)') called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 498
Foswiki::UI::_execute('Foswiki::Request=HASH(0x8822750)', 'CODE(0x8822600)', 'jsonrpc', 1) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 324
Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x8822750)') called at /var/www/foswiki/distro/core/lib/Foswiki/Engine/CGI.pm line 98
Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x83a6e50)') called

ABOUT TO EVAL { 'Foswiki::Func' => [ '&getSkin', #'&getUrlHost', '&getScriptUrl', ] }
RETURNING { 'Foswiki::Func' => [ '&getSkin', #'&getUrlHost', '&getScriptUrl', ] }

Completely wipes out 1.2 configure. This needs some fixup before 1.2 release! I can't get into configure with this extension installed. Points to some issues in the Config.spec parser. It's become more brittle.

-- GeorgeClark - 24 Dec 2014

The issue is that the Config.spec shipped with PerlPlugin includes some embedded comments that don't eval. We could probably stand to handle this a bit more gracefully in configure. Committed fix to PerlPlugin. This task can "No Action" ... configure 1.2 displays the settings in the right location.

-- GeorgeClark - 26 Dec 2014

I designed configure on the principle that Config.spec's would be tested by extension authors.

We could protect against Config.spec authors shooting themselves in the foot. That would require eval of all DISPLAY_IF and CHECK expressions as well, which requires a browser. I don't know any way to do that automatically.

-- CrawfordCurrie - 27 Dec 2014

In this case there is an eval in getspec that is trapped and passed to reporter as an ERROR. But for some reason it is trapped and reported as a jsonrpc crash and not as an item/value error. In this block in Foswiki::Configure::LoadSpec::parse()

            # check it's a valid perl expression, ignoring uninitialised
            # variable warnings inside strings.
            no warnings;
            $value =~ /^(.*)$/;
            eval $1;
            $reporter->ERROR("$context: Cannot eval value '$value': $@") if $@;
            $value = $1;
            use warnings;

I tried a local sig die and a few other things but could not prevent the error from crashing jsonrpc. The eval is completely ineffective. I wonder if this is related to the generally buggy / magical aspects of try/catch. If the reporter was actually trapping the error and allowing configure to proceed, we'd be in much better shape.

-- GeorgeClark - 27 Dec 2014

Dug just a little further. On 1.1.9, the comments in the Config.spec perl definition appear to be silently filtered out. I installed PerlPlugin from Extensions web on my 1.1.9 test system to no ill effect. Configure apparently doesn't eval the perl, or if it does, it cleans it first.

I've now thoroughly hijacked this task The original problem was a "no action" though. On 1.1.9 the settings are in the correct location.

The issue for 1.2 is that upgraders using existing extensions are going to have a configure that doesn't load. So we've retroactively added bullets to their gun, and aimed it at their feet for them. The config.spec files were tested on 1.1.x, and worked there. So we've tightened the spec. That's not a bad thing, but it would be nice if we were graceful about it.

-- GeorgeClark - 27 Dec 2014

The only way I can see that would fail is if $1 is wrong, because the regex failed for some reason (e.g. $value was undef)

-- CrawfordCurrie - 27 Dec 2014

$Foswiki::cfg{Plugins}{PerlPlugin}{Share} = {
    'Foswiki::Func' => [
        '&getSkin',
        #'&getUrlHost',
        '&getScriptUrl',
    ]
 };

 sub _loadSpecsFrom {
@@ -343,8 +344,8 @@ sub parse {
             while ( $value !~ s/\s*;\s*$// ) {
                 my $cont = <$fh>;
                 last unless defined $cont;
                chomp $cont;
                $cont =~ s/^\s+/ /;
                 unless ( $cont =~ /^#/ ) {
                     $value .= $cont;
                 }

{'FIELD' => {'name' => {'index' => 1,'type' => 'VARCHAR(128)'},'tid' => {'type' => 'INT'},'title' => {'type' => 'VARCHAR(256)'},'value' => {'type' => 'TEXT'}},'FILEATTACHMENT' => {'attr' => {'type' => 'VARCHAR(32)'},'comment' => {'truncate_to' => 512,'type' => 'VARCHAR(512)'},'date' => '_DATE','name' => {'index' => 1,'type' => 'VARCHAR(256)'},'path' => {'type' => 'VARCHAR(256)'},'size' => {'type' => 'VARCHAR(32)'},'tid' => {'type' => 'INT'},'user' => '_USERNAME','version' => {'type' => 'VARCHAR(32)'}},'FORM' => {'name' => {'index' => 1,'type' => 'VARCHAR(256)'},'tid' => {'type' => 'INT'}},'PREFERENCE' => {'name' => {'index' => 1,'type' => 'VARCHAR(64)','unique' => 'onepref'},'tid' => {'type' => 'INT','unique' => 'onepref'},'type' => {'type' => 'VARCHAR(32)'},'value' => '_DEFAULT'},'TOPICINFO' => {'author' => '_USERNAME','comment' => {'type' => 'VARCHAR(512)'},'date' => '_DATE','encoding' => {'type' => 'VARCHAR(32)'},'format' => {'type' => 'VARCHAR(32)'},'reprev' => {'type' => 'VARCHAR(32)'},'rev' => {'type' => 'VARCHAR(32)'},'tid' => {'type' => 'INT','unique' => 'onetopicinfo'},'version' => {'type' => 'VARCHAR(256)'}},'TOPICMOVED' => {'by' => {'type' => 'VARCHAR(256)'},'date' => '_DATE','from' => {'type' => 'VARCHAR(256)'},'tid' => {'type' => 'INT'},'to' => {'type' => 'VARCHAR(256)'}},'TOPICPARENT' => {'name' => {'index' => 1,'type' => 'VARCHAR(256)'},'tid' => {'type' => 'INT','unique' => 'oneparent'}},'_DATE' => {'type' => 'VARCHAR(32)'},'_DEFAULT' => {'type' => 'TEXT'},'_USERNAME' => {'index' => 1,'type' => 'VARCHAR(64)'},'metatypes' => {'name' => {'primary' => 1,'type' => 'VARCHAR(63)'}},'topic' => {'name' => {'index' => 1,'type' => 'VARCHAR(128)','unique' => 'webtopic'},'raw' => '_DEFAULT','text' => '_DEFAULT','tid' => {'primary' => 1,'type' => 'INT'},'web' => {'index' => 1,'type' => 'VARCHAR(256)','unique' => 'webtopic'}}}

$Foswiki::cfg{Extensions}{DBIStoreContrib}{Schema} = {
    _DEFAULT => { type => 'TEXT' },
    _USERNAME => { type => 'VARCHAR(64)', index => 1 },
    _DATE => { type => 'VARCHAR(32)' },
    topic => {
        _level => 0,
        tid  => { type => 'INT', primary => 1 }
        web  => { type => 'VARCHAR(256)', index => 1 },
        name => { type => 'VARCHAR(128)', index => 1 },
        text => '_DEFAULT',
        raw  => '_DEFAULT'
        },
    metatypes => {
        _level => 0,
        name => { type => 'VARCHAR(63)', index => 1 },
        },
    TOPICINFO => {
        _level => 1,
        author => '_USERNAME',
        version => { type => 'VARCHAR(256)' },
        date => '_DATE',
        format => { type => 'VARCHAR(32)' },
        reprev => { type => 'VARCHAR(32)' },
        rev => { type => 'VARCHAR(32)' },
        comment => { type => 'VARCHAR(512)' },
        encoding => { type => 'VARCHAR(32)' },
    },
    TOPICMOVED => {
        _level => 1,
        from => { type => 'VARCHAR(256)' },
        to => { type => 'VARCHAR(256)' },
        by => { type => 'VARCHAR(256)' },
        date => '_DATE',
    },
    TOPICPARENT => {
        _level => 1,
        name => { type => 'VARCHAR(256)', index => 1 },
    },
    FILEATTACHMENT => {
        _level => 1,
        name => { type => 'VARCHAR(256)', index => 1 },
        version => { type => 'VARCHAR(32)' },
        path => { type => 'VARCHAR(256)' },
        size => { type => 'VARCHAR(32)' },
        date => '_DATE',
        user => '_USERNAME',
        comment => { type => 'VARCHAR(512)', truncate_to => 512 },
        attr => { type => 'VARCHAR(32)' }
    },
    FORM => {
        _level => 1,
        name => { type => 'VARCHAR(256)', index => 1 },
    },

• Press "Reset" - field gains structure, and shows up as modified
• Save Configuration
• Field returns to unstructured on-line.

• Value before reset:

  [    
      {
          'multivalued' ... 

• Value after reset:

   [    {
          'multivalued'...