Item13455: PlainFileStore fails to clean up ,pfv directories for utf-8 topics

pencil
Priority: Normal
Current State: Closed
Released In: 1.2.0
Target Release: n/a
Applies To: Engine
Component: PlainFileStoreContrib
Branches: master
Reported By: GeorgeClark
Waiting For:
Last Change By: CrawfordCurrie
The Fn_SEARCH tests are extremely noisy on my system. I've tried two different perl versions, 5.14.2 (ubuntu) and 5.20.2 (perlbrew)

I've added debug messages into the _rmtree function in PlainFileStore. Also lists the directory on failure. Seems to show that the following were left behind:
  • OkÃTopic,pfv
  • SomeOtherÃÃÃTopic,pfv

exporting FOSWIKI_ASSERTS=1 for extra checking; disable by exporting FOSWIKI_ASSERTS=0
Options: -clean
Assert checking on 1
DEVELOPER WARNING: taint mode could not be enabled. Is Taint::Runtime installed?
Starting CWD is /var/www/foswiki/distro/core/test/unit 
Running Fn_SEARCH
        Fn_SEARCH::test_nofinalnewline
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/TestTopicSEARCH,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/SomeOtherÃÃÃTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/.changes
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/WebPreferences.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkBTopic,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/InvisibleTopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkATopic.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH/OkÃTopic.txt
total 16
drwxr-xr-x  4 gac gac   4096 Jun  9 21:05 .
drwxr-xr-x 15 gac users 4096 Jun  9 21:05 ..
drwxr-xr-x  2 gac gac   4096 Jun  9 21:05 OkÃTopic,pfv
drwxr-xr-x  2 gac gac   4096 Jun  9 21:05 SomeOtherÃÃÃTopic,pfv
Unexpected exception while removing web TemporarySEARCHTestWebSEARCH
PlainFile: Failed to delete /var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH:  at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 1476.
 at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 45
        FoswikiTestCase::__ANON__('PlainFile: Failed to delete /var/www/foswiki/distro/core/data...') called at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 1476
        Foswiki::Store::PlainFile::_rmtree('/var/www/foswiki/distro/core/data/TemporarySEARCHTestWebSEARCH') called at /var/www/foswiki/distro/core/lib/Foswiki/Store/PlainFile.pm line 791
        Foswiki::Store::PlainFile::remove('Foswiki::Store::PlainFile=HASH(0x4289ee8)', 'BaseUserMapping_666', 'Foswiki::Meta=HASH(0x44f9898)', undef) called at /var/www/foswiki/distro/core/lib/Foswiki/Meta.pm line 2618
        Foswiki::Meta::removeFromStore('Foswiki::Meta=HASH(0x44f9898)') called at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 821
        FoswikiTestCase::__ANON__() called at /usr/share/perl5/Error.pm line 416
        eval {...} called at /usr/share/perl5/Error.pm line 408
        Error::subs::try('CODE(0x44e0db0)', 'HASH(0x45c61e8)') called at /var/www/foswiki/distro/core/test/unit/FoswikiTestCase.pm line 827
        FoswikiTestCase::removeWebFixture('Fn_SEARCH=HASH(0x3732620)', 'Foswiki=HASH(0x425cea8)', 'TemporarySEARCHTestWebSEARCH') called at /var/www/foswiki/distro/core/test/unit/FoswikiFnTestCase.pm line 129
        FoswikiFnTestCase::tear_down('Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH::test_nofinalnewline') called at /var/www/foswiki/distro/core/test/unit/Fn_SEARCH.pm line 194
        Fn_SEARCH::tear_down('Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH::test_nofinalnewline') called at /var/www/foswiki/distro/core/lib/Unit/TestRunner.pm line 604
        Unit::TestRunner::runOne('Unit::TestRunner=HASH(0x2606e48)', 'Fn_SEARCH=HASH(0x3732620)', 'Fn_SEARCH', 'test_nofinalnewline') called at /var/www/foswiki/distro/core/lib/Unit/TestRunner.pm line 163
        Unit::TestRunner::start('Unit::TestRunner=HASH(0x2606e48)', 'Fn_SEARCH::test_nofinalnewline') called at ../bin/TestRunner.pl line 143.

Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/.changes
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WebPreferences.txt
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/WikiUsers,pfv/1
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv
Calling _rmtree for /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv/1.m
Processing /var/www/foswiki/distro/core/data/TemporarySEARCHUsersWeb/ScumBag,pfv/1

Unit test run Summary:
 - Tested with perl v5.14.2 from /usr/bin/perl

All tests passed (1)
1..21

-- GeorgeClark - 10 Jun 2015

The following patch fixes the failures, however I don't think it's right. I think the issue is that _rmtree() is encoding something that is already encoded, so the _decode removes the double encoding. However removing the _encode at the top of _rmtree() breaks a lot of things.

diff --git a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
index bea2b56..1ed9463 100644
--- a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
+++ b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
@@ -1449,6 +1449,8 @@ sub _rmtree {
             $entry = "$root/$1";
             if ( -d $entry ) {
                 _rmtree($entry);
+                _rmtree(_decode($entry)) if -e $entry;
+                die "STILL THERE" if -e $entry;
             }
             elsif ( !unlink($entry) && -e $entry ) {
                 my $mess = 'PlainFile: Failed to delete file '

-- GeorgeClark - 13 Jun 2015

Another fix also appears to work fine. Move the call to _encode up to the callers of _rmtree, so that there is no encoding of recursive calls. I assume the issue is that the directory read returns bytes, so it does not need further encoding.
diff --git a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
index bea2b56..569d103 100644
--- a/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
+++ b/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
@@ -782,14 +782,14 @@ sub remove {
 
         # Topic or attachment
         _unlink( _latestFile( $meta, $attachment ) );
-        _rmtree( _historyDir( $meta, $attachment ) );
-        _rmtree( _getPub($meta) ) unless ($attachment);    # topic only
+        _rmtree( _encode(_historyDir( $meta, $attachment )) );
+        _rmtree( _encode(_getPub($meta) ) ) unless ($attachment);    # topic only
     }
     else {
 
         # Web
-        _rmtree( _getData($meta) );
-        _rmtree( _getPub($meta) );
+        _rmtree( _encode(_getData($meta)) );
+        _rmtree( _encode(_getPub($meta)) );
     }
 
     return unless ( $Foswiki::Store::STORE_FORMAT_VERSION < 1.2 );
@@ -1438,7 +1438,7 @@ sub _mkPathTo {
 
 # Remove an entire directory tree. $root must be encoded.
 sub _rmtree {
-    my $root = _encode(shift);
+    my $root = shift;
     my $D;
     if ( opendir( $D, $root ) ) {
 

ItemTemplate edit

Summary PlainFileStore fails to clean up ,pfv directories for utf-8 topics
ReportedBy GeorgeClark
Codebase trunk
SVN Range
AppliesTo Engine
Component PlainFileStoreContrib
Priority Normal
CurrentState Closed
WaitingFor
Checkins distro:2ece92b9f13b
TargetRelease n/a
ReleasedIn 1.2.0
CheckinsOnBranches master
trunkCheckins
masterCheckins distro:2ece92b9f13b
ItemBranchCheckins
Release01x01Checkins
Topic revision: r6 - 15 Jun 2015, CrawfordCurrie
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy