Item13466: Foswiki crash in logger if installed in a directory named using utf-8 characters

• install 1.2 beta2 in an utf 8 directory (directory which contains utf8 characters and you will get an logger crash.
• for my test I've used /var/www/foswikiBêta2, but in practice, it could happen for any installation which used an utf8 name e.g. : /var/www//foswiki
• copy/paste error

    Software error:
     
    ERROR: Could not open logfile /var/www/foswikiBêta2/working/logs/error.log for write. Your admin should 'configure' now and fix the errors!
     at /usr/share/perl5/CGI/Carp.pm line 353.
            CGI::Carp::realdie("ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /usr/share/perl5/CGI/Carp.pm line 443
            CGI::Carp::die("ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /var/www/foswikiBêta2/lib/Foswiki/Logger/PlainFile.pm line 145
            Foswiki::Logger::PlainFile::log(Foswiki::Logger::PlainFile=HASH(0x98bec78), "warning", "ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 475
            Foswiki::UI::__ANON__(Error::Simple=HASH(0x987d2e4), SCALAR(0x8be20e0)) called at /usr/share/perl5/Error.pm line 340
            eval {...} called at /usr/share/perl5/Error.pm line 330
            Error::subs::run_clauses(HASH(0x94fca54), "ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"..., undef, ARRAY(0x8bea90c)) called at /usr/share/perl5/Error.pm line 427
            Error::subs::try(CODE(0x8a96e30), HASH(0x94fca54)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 500
            Foswiki::UI::_execute(Foswiki::Request=HASH(0x9514514), CODE(0x8edc174), "view", 1) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 326
            Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x9514514)) called at /var/www/foswikiBêta2/lib/Foswiki/Engine/CGI.pm line 99
            Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x8f6ff94)) called at /var/www/foswikiBêta2/bin/view line 29.
     
    For help, please send mail to the webmaster (webmaster@localhost), giving this error message and the time and date of the error.
    Software error:
     
    [Sat Jun 20 15:23:00 2015] view: ERROR: Could not open logfile /var/www/foswikiBêta2/working/logs/error.log for write. Your admin should 'configure' now and fix the errors!
    [Sat Jun 20 15:23:00 2015] view:  at /usr/share/perl5/CGI/Carp.pm line 353.
    [Sat Jun 20 15:23:00 2015] view:        CGI::Carp::realdie("ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /usr/share/perl5/CGI/Carp.pm line 443
    [Sat Jun 20 15:23:00 2015] view:        CGI::Carp::die("ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /var/www/foswikiBêta2/lib/Foswiki/Logger/PlainFile.pm line 145
    [Sat Jun 20 15:23:00 2015] view:        Foswiki::Logger::PlainFile::log(Foswiki::Logger::PlainFile=HASH(0x98bec78), "warning", "ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"...) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 475
    [Sat Jun 20 15:23:00 2015] view:        Foswiki::UI::__ANON__(Error::Simple=HASH(0x987d2e4), SCALAR(0x8be20e0)) called at /usr/share/perl5/Error.pm line 340
    [Sat Jun 20 15:23:00 2015] view:        eval {...} called at /usr/share/perl5/Error.pm line 330
    [Sat Jun 20 15:23:00 2015] view:        Error::subs::run_clauses(HASH(0x94fca54), "ERROR: Could not open logfile /var/www/foswikiB\x{c3}\x{aa}ta2/working/"..., undef, ARRAY(0x8bea90c)) called at /usr/share/perl5/Error.pm line 427
    [Sat Jun 20 15:23:00 2015] view:        Error::subs::try(CODE(0x8a96e30), HASH(0x94fca54)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 500
    [Sat Jun 20 15:23:00 2015] view:        Foswiki::UI::_execute(Foswiki::Request=HASH(0x9514514), CODE(0x8edc174), "view", 1) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 326
    [Sat Jun 20 15:23:00 2015] view:        Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x9514514)) called at /var/www/foswikiBêta2/lib/Foswiki/Engine/CGI.pm line 99
    [Sat Jun 20 15:23:00 2015] view:        Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x8f6ff94)) called at /var/www/foswikiBêta2/bin/view line 29.
     at /usr/share/perl5/CGI/Carp.pm line 353.
            CGI::Carp::realdie("[Sat Jun 20 15:23:00 2015] view: ERROR: Could not open logfil"...) called at /usr/share/perl5/CGI/Carp.pm line 450
            CGI::Carp::die(Error::Simple=HASH(0x989c53c)) called at /usr/share/perl5/Error.pm line 184
            Error::throw(Error::Simple=HASH(0x989c53c)) called at /usr/share/perl5/Error.pm line 433
            Error::subs::try(CODE(0x8a96e30), HASH(0x94fca54)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 500
            Foswiki::UI::_execute(Foswiki::Request=HASH(0x9514514), CODE(0x8edc174), "view", 1) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 326
            Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x9514514)) called at /var/www/foswikiBêta2/lib/Foswiki/Engine/CGI.pm line 99
            Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x8f6ff94)) called at /var/www/foswikiBêta2/bin/view line 29
     
    For help, please send mail to the webmaster (webmaster@localhost), giving this error message and the time and date of the error.

-- GuilainCabannes - 20 Jun 2015

I've recreated the issue. Installed into directory: /var/www/letiště And bootstrap won't run. Slightly different crash:

[Sat Jun 20 10:06:20 2015] view: Use of uninitialized value $text in substitution (s///) at /var/www/letiÅ¡tÄlib/Foswiki/Plugins/InterwikiPlugin.pm line 90.

Note that Apache 2.4 appears to serve pub files correctly from a utf-8 path.

-- GeorgeClark - 20 Jun 2015

Crawford, I didn't mark this for 1.2. I suspect this is another can of worms, so probably better to just document a restriction like we do for Windows. Installation path cannot contain utf-8 characters.

-- GeorgeClark - 20 Jun 2015

Running a CLI install tools/configure -save and then things appear to be operational. So maybe an issue in bootstrap rather than mainline foswiki.

-- GeorgeClark - 20 Jun 2015

I have try tools/configure -save without any success

It appears that the different component deal in different manner the utf8 path. Someone have the "\x{ea}" other have "ê".

CGI::Carp::realdie("ERROR: Could not open logfile /var/www/foswikiB\x{ea}ta2/working/l"...) called at /usr/share/perl5/CGI/Carp.pm line 443
   CGI::Carp::die("ERROR: Could not open logfile /var/www/foswikiB\x{ea}ta2/working/l"...) called at /var/www/foswikiBêta2/lib/Foswiki/Logger/PlainFile.pm line 145
   Foswiki::Logger::PlainFile::log(Foswiki::Logger::PlainFile=HASH(0xad8d02c), "warning", "ERROR: Could not open logfile /var/www/foswikiB\x{ea}ta2/working/l"...) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 475
   Foswiki::UI::__ANON__(Error::Simple=HASH(0xad4a814), SCALAR(0xa11f23c)) called at /usr/share/perl5/Error.pm line 340
   eval {...} called at /usr/share/perl5/Error.pm line 330
   Error::subs::run_clauses(HASH(0xa8eb74c), "ERROR: Could not open logfile /var/www/foswikiB\x{ea}ta2/working/l"..., undef, ARRAY(0xa127aa8)) called at /usr/share/perl5/Error.pm line 427
   Error::subs::try(CODE(0x9fd3ec8), HASH(0xa8eb74c)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute(Foswiki::Request=HASH(0xa8cd080), CODE(0xa828c84)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest(Foswiki::Request=HASH(0xa8cd080)) called at /var/www/foswikiBêta2/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0xa4f5130)) called at /var/www/foswikiBêta2/bin/configure line 29.

Totaly agree to defer to an other release, because It could be quickly avoid by adding restriction information on release note.

-- GuilainCabannes - 20 Jun 2015

Digging a bit more, there are a couple of areas where this is broken.

• Bootstrap doesn't convert from bytes to utf-8 when bootstrapping the paths.
• If that is fixed, then tools/configure breaks.
• The Foswiki::Configure::FileUtil path checkers don't handle Webnames in utf-8 when reading the data or pub directories.

Crawford, I'm checking in a fix, but it needs review.

-- GeorgeClark - 21 Jun 2015

REVERTED the bootstrap and tools/configure part of my checkin. It's still crashing after a bootstrap.

[Sat Jun 20 23:20:45 2015] view: [Sat Jun 20 23:20:45 2015] view: Malformed UTF-8 character (fatal) at /usr/share/perl/5.20/Carp.pm line 306.
[Sat Jun 20 23:20:45 2015] view:  at /usr/share/perl5/CGI/Carp.pm line 353.
[Sat Jun 20 23:20:45 2015] view:        CGI::Carp::realdie("[Sat Jun 20 23:20:45 2015] view: Malformed UTF-8 character (f"...) called at /usr/share/perl5/CGI/Carp.pm line 450
[Sat Jun 20 23:20:45 2015] view:        CGI::Carp::die(Error::Simple=HASH(0x229b318)) called at /usr/share/perl5/Error.pm line 184
[Sat Jun 20 23:20:45 2015] view:        Error::throw(Error::Simple=HASH(0x229b318)) called at /usr/share/perl5/Error.pm line 433
[Sat Jun 20 23:20:45 2015] view:        Error::subs::try(CODE(0xa427e0), HASH(0x1ab8620)) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 500
[Sat Jun 20 23:20:45 2015] view:        Foswiki::UI::_execute(Foswiki::Request=HASH(0x19e7df0), CODE(0x17056d8), "view", 1) called at /var/www/foswikiBêta2/lib/Foswiki/UI.pm line 326
[Sat Jun 20 23:20:45 2015] view:        Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x19e7df0)) called at /var/www/foswikiBêta2/lib/Foswiki/Engine/CGI.pm line 99
[Sat Jun 20 23:20:45 2015] view:        Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x139ce10)) called at view line 29
[Sat Jun 20 23:27:05.540158 2015] [cgid:error] [pid 6563:tid 139867044357888] (32)Broken pipe: [client 127.0.0.1:60107] AH02550: Failed to flush CGI output to client

-- GeorgeClark - 21 Jun 2015

The character we were using in the path to test was ê, which is unicode 0xEA. Annoyingly, that's a high-bit character. So, when the bootstrap initialised using it, it was able to write the LSC even though it hadn't opened LSC with an encoding - the EA was a valid byte. When LSC was opened for read, it was opened with encoding(utf8) and EA is not a valid utf8 character, hence the "Malformed UTF-8 character"

The solution is to open LSC encoding(utf-8) when writing it. Plus, a better test is to use a wide-byte character, e.g. gぃt

-- Main.CrawfordCurrie - 24 Jun 2015 - 09:18

Because of the AsciiOnlyLSC™ :).

Anyway, here is an easy workaround using the command line tools/configure. The steps, what I tried:

curl -L http://sourceforge.net/projects/foswiki/files/foswiki/1.2.0_Beta_2/Foswiki-1.2.0_Beta_2.tgz > Foswiki-1.2.0_Beta_2.tgz
mkdir žížala || exit 1
cd žížala
tar xvzf ../Foswiki-1.2.0_Beta_2.tgz 
mv Foswiki-1.2.0_Beta_2 fw12b2   #only for shortening
cd fw12b2/tools
perl configure -noprompt \
    -set '{DefaultUrlHost}=http://localhost:8000' \
    -set '{ScriptUrlPath}=/bin' \
    -set '{ScriptUrlPaths}{view}=' \
    -set '{PubUrlPath}=/pub' \
    -set '{Password}=q' \
    -set '{Store}{Implementation}=Foswiki::Store::PlainFile' \
    -set '{Store}{Encoding}=utf-8' \
    -save
touch ../data/.htpasswd
./lighttpd.pl -p 8000

Check the LSC (lib/LocalSite.cfg), it contains something like:
$Foswiki::cfg{DataDir} = '/me/fw/sites/žížala/fw12b2/data'; and so on, but the Foswiki will read it OK,

• open http://localhost:8000 and it just works...
• of course the žížala directory is created using utf8 shell (not iso1)

Convert the LSC to ascii-only

• in the browser go to http://localhost:8000/bin/configure
• because unfortunately the configure doesn't have plain SAVE (without changing anything - again one strange "feature")
• change anything, for example in the Security -> Sessions change the
  {Sessions}{ExpireAfter}=21601 . You can change it back anytime.
• save one change
• and you get an AsciiOnlyLSC (check your LSC, now contains a line such
  $Foswiki::cfg{DataDir} = "/me/fw/sites/\x{17e}\x{ed}\x{17e}ala/fw12b2/data";
• done - working(?) Foswiki on utf8 path. (At least the basic - don't checked any Extension).

-- JozefMojzis - 24 Jun 2015

LSC cannot be ascii-only, because nothing else is. And it isn't, any more, it's utf-8 encoded for 1.2,

I believe I have fixed the outstanding problem.

-- Main.CrawfordCurrie - 24 Jun 2015 - 10:14

Great! I missed it. So, it works. Thank you!

Just installed a foswiki into a path: /me/fw/sites/i♥foswiki

-- JozefMojzis - 24 Jun 2015

Commenting is disabled while not logged in