Item13573: MultiTopicSavePlugin fails in 2.0 because of some incompatible changes in charsets ≤
Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a
I use the plugin
MultiTopicSavePlugin extensively (Made it myself) and it works very well in 1.1.9
I am running on a Git master clone. I need to run the server so it is compatible with all my existing topics so I have chosen iso-8859-1 instead of UTF8. There is no way I can start converting 100000s of topics to UTF8 without risk of screwing up business critical information.
I also run with RCS Wrap.
The plugin fails when saving.
This is the error reported.
This is a real show stopper for me and I have no clue how to fix it in the plugin.
I am releasing a new version of
MultiTopicSavePlugin now and it will not work with 2.0. Any fixes need to be compatible with 1.1.9 also
Even of this is a plugin related bug the problem is that 2.0 is not compatible with existing plugins. Something related to the UTF8 changes you did is broken.
Note that the plugin uses the rest interface. And I am using only the API in the plugin.
ERROR: (500) Internal server error - "\x{2264}" does not map to iso-8859-1 at /usr/lib64/perl5/vendor_perl/Encode.pm line 160.
at /usr/lib64/perl5/vendor_perl/Encode.pm line 160.
Encode::encode('iso-8859-1', '%META:TOPICINFO{author="c12179" comment="reprev" date="143810...', 1) called at /var/www/foswikidev/core/lib/Foswiki/Store.pm line 276
Foswiki::Store::encode('%META:TOPICINFO{author="c12179" comment="reprev" date="143810...') called at /var/www/foswikidev/core/lib/Foswiki/Store/Rcs/Store.pm line 429
Foswiki::Store::Rcs::Store::saveTopic('Foswiki::Store::RcsWrap=HASH(0x2224440)', 'Foswiki::Meta=HASH(0x32223f0)', 'c12179', 'HASH(0x1ca4df0)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2135
Foswiki::Meta::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
Error::subs::try('CODE(0x36cea20)', 'HASH(0x36cf020)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2152
Foswiki::Meta::saveAs('Foswiki::Meta=HASH(0x32223f0)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 1999
Foswiki::Meta::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
Error::subs::try('CODE(0x3475d48)', 'HASH(0x36c9d08)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2003
Foswiki::Meta::save('Foswiki::Meta=HASH(0x32223f0)') called at /var/www/foswikidev/core/lib/Foswiki/Func.pm line 2115
Foswiki::Func::saveTopic('Station/Track', 'HwSR00001151', 'Foswiki::Meta=HASH(0x3460ac8)', '---++ Status Updates\x{a}\x{a}_Assessor and Assignee_\x{a}\x{a}\x{a}---++++ Statu...') called at /var/www/foswikidev/core/lib/Foswiki/Plugins/MultiTopicSavePlugin.pm line 704
Foswiki::Plugins::MultiTopicSavePlugin::restMultiTopicSave('Foswiki=HASH(0x2223b88)', 'MultiTopicSavePlugin', 'multitopicsave', 'Foswiki::Response=HASH(0x2223cc0)') called at /var/www/foswikidev/core/lib/Foswiki/Func.pm line 768
Foswiki::Func::__ANON__('Foswiki=HASH(0x2223b88)', 'MultiTopicSavePlugin', 'multitopicsave', 'Foswiki::Response=HASH(0x2223cc0)') called at /var/www/foswikidev/core/lib/Foswiki/UI/Rest.pm line 275
Foswiki::UI::Rest::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
Error::subs::try('CODE(0x3425ea8)', 'HASH(0x32222b8)') called at /var/www/foswikidev/core/lib/Foswiki/UI/Rest.pm line 290
Foswiki::UI::Rest::rest('Foswiki=HASH(0x2223b88)') called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 374
Foswiki::UI::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
Error::subs::try('CODE(0x13edf00)', 'HASH(0x23158c0)') called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 500
Foswiki::UI::_execute('Foswiki::Request=HASH(0x13f57a8)', 'CODE(0x22ffb38)', 'rest', 1) called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 326
Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x13f57a8)') called at /var/www/foswikidev/core/lib/Foswiki/Engine/CGI.pm line 99
Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x1a858a8)') called at /var/www/foswikidev/core/bin/rest line 29.
--
KennethLavrsen - 28 Jul 2015
Update. The bug is triggered by having an html entity ≤ in a formfield. This content gets converted to UTF8. That does not happen in 1.1.9. It seems to be a rest interface related problem. I can edit and save the same topic just fine in the normal edit/save sequency.
--
KennethLavrsen - 28 Jul 2015
The content in a formfield is exactly this
Condition ---- RF_CPLD_RX_FE_SEL[1,2]
Rx freq ≤ FE_RANGE1 ---- 11
FE_RANGE1 < Rx freq ≤ FE_RANGE2 ---- 10
FE_RANGE2 < Rx freq ≤ FE_RANGE3 ---- 01
FE_RANGE3 < RX freq ---- 00
So when sending this through the rest interface the ≤ gets translated to UTF somewhere and then the Encode for iso fails
--
KennethLavrsen - 28 Jul 2015
See also related
Item13575
--
GeorgeClark - 28 Jul 2015
The update that Clarke did in
Item13575 changed the behaviour of this bug. The plugin can now save topics with html entities that would have become utf8. BUT it is still not working. Everytime the plugin compares the old value with the new submitted it sees a difference so it saves the topic even though you did not touch it. That is a big problem.
The plugin reads a fields value with and uses this to display
my ( $meta, undef ) = Foswiki::Func::readTopic( $web, $topic );
my $value = $meta->get( 'FIELD', $field );
my $returnvalue = defined $value ? $value->{'value'} : '';
When we compare the new value with the old, the old value is read with same meta->get
my $fieldhashref = $meta->get( 'FIELD', $fieldName );
$oldValue = $fieldhashref ?
$meta->get( 'FIELD', $fieldName )->{'value'} :
'';
Later the value is put back with
$meta->putKeyed( 'FIELD', { name => $fieldName, value => $value } );
So the conversion must likely happens when the user submits the form via the rest interface.
the place the submit value is picked up is
my @fieldValues = $query->multi_param($key);
For this plugin to ever work, The plugin needs to get the unchanged content in all 3 cases.
$meta->get, and $query->multi_param must return the content 100% identical of what I saved with $meta->putKeyed. If a conversion happens it must be 100% two way transparent.
--
KennethLavrsen - 29 Jul 2015
After a lot of analysis it is clear that the my @fieldValues = $query->multi_param($key); is returning encoded values. HTML entities get encoded to a value like \xe2\x89\xa4 for a ≤
The $meta->get( 'FIELD', $fieldName ); returns lean ≤
The $meta->putKeyed( 'FIELD', { name => $fieldName, value => $value } ); stores cleanly ≤
The Browser receives a clean ≤ from the server. I can see that by viewing sources.
Something is goofy with @fieldValues = $query->multi_param($key);
--
KennethLavrsen - 29 Jul 2015
Adding a little test case
This test case is simple. Click the edit. Note that there is a %le; in the Summary field of this Task. If you leave it untouched, the plugin should say that nothing changed and should not change this topic. When it fails it says that 1 topic is changed.
--
KennethLavrsen - 29 Jul 2015
Note that this test case does not fails here on foswiki.org for some reason. Perl version? UTF8 vs ISO?
--
KennethLavrsen - 29 Jul 2015
In 1.1.9 the ≤ gets converted to ≤ when using the plugin
This gets saved and from then on ≤ gets converted to itself and I do not see these ghost changes more than once. That is why it works. But there is still a conversion happening.
--
KennethLavrsen - 29 Jul 2015
Using the test case in this topic, along with firebug & wireshark, the culprit is the browser. I updated the summary to contain the
≤
entity. The
MultiTopicSavePlugin created a textarea containing:
<textarea class='foswikiTextarea' cols='80' rows='1' name='multitopicsavefield{Tasks.Item13573}{Summary
}'>MultiTopicSavePlugin fails in 2.0 because of some incompatible changes in charsets ≤</textarea
>
Upon clicking Save, the POST contained:
validation_key=78846875ccd9765b22f5fcf8a5a43922&redirectweb=Tasks&redirecttopic=Item13573&topic=Tasks
.Item13573&multitopicsavefield%7BTasks.Item13573%7D%7BSummary%7D=MultiTopicSavePlugin+fails+in+2.0+because
+of+some+incompatible+changes+in+charsets++%E2%89%A4
So the browser is converting the named entity into the hex entity.
--
GeorgeClark - 30 Jul 2015
Here is the "round trip"
- the original form data contains literal
≤
- ->> Sent to browser as
</textarea> ≤ </textarea>
- User presses submit, touching nothing
- <<- POST contains %E2%89%A4
- Foswiki automatically converts %E2%89%A4 into the unicode ≤
- Plugin compares ≤ to ≤ and fails
We can't reliably convert the posted ≤ into an entity, we don't really know if the original topic contained the named or numeric entity. But what you can do is convert the named entities into unicode characters. So continuing on...
- Plugin does HTML::Entities::decode_entities(original formfield source)
- Now plugin compares ≤ to ≤ and they should match.
And if they don't match, and plugin saves the topic:
- For a utf-8 store, the ≤ is written to the store as a unicode character.
- For an iso* store, after the patch to Item13575, the ≤ causes a utf8 exception, and HTML::Entities converts it to the named ≤ entity.
- Prior to Item13757, Store would crash if the {Store}{Encoding} did not contain that character.
--
GeorgeClark - 30 Jul 2015
This task seems to be fixed. Just ran the above testcase again, and it did not save the topic, which is the behaviour you want, though I'm testing on a uft-8 store, Not sure what would happen on an iso-8859 store.
--
GeorgeClark - 24 Nov 2015
Can this be closed?
--
GeorgeClark - 24 Dec 2015