You are here: Foswiki>Tasks Web>Item13566 (26 Sep 2016, MichaelDaum)Edit Attach

Item13566: Speed-aware reading encoded files

Priority: Enhancement
Current State: Confirmed
Released In: n/a
Target Release: n/a
Applies To: Engine
Component: Performance, Unicode
Reported By: JozefMojzis
Waiting For:
Last Change By: MichaelDaum
Benchmarked different file-reading and decoding methods, and got some surprising results.

So asked an question in the Stackoverflow.

The answer pointed me to the module CPAN:Unicode::UTF8, it allows reading files 5 times faster as using Encode. The question contains some benchmarking code too.

Therefore, (and learned from the CGI-problem lesson too), IMHO Foswiki shouldn't use any CPAN modules directly, but thru its own interface modules, like Foswiki::Encode - instead of the plain perl's Encode:. Such approach could allow easily modify, replace the backend implementation, e.g. in this case the file-reading routine could be:

if( the_store_is_utf8 ) {
    read RAW file and use Unicode::UTF8 for the speed gain
} else {
    read the file using Encode

-- JozefMojzis - 27 Jul 2015

Setting this to confirmed. But does need more investigation.

-- GeorgeClark - 25 Sep 2016

Jozef, which perl version have you been benchmarking. Note that anything befor 5.22 is not worth it. I just looked at Unicode::UTF8's activity on github. ... which logs most recent commit back in 2013. Hm.

-- MichaelDaum - 26 Sep 2016

ItemTemplate edit

Summary Speed-aware reading encoded files
ReportedBy JozefMojzis
Codebase trunk
SVN Range
AppliesTo Engine
Component Performance, Unicode
Priority Enhancement
CurrentState Confirmed
TargetRelease n/a
ReleasedIn n/a
Topic revision: r3 - 26 Sep 2016, MichaelDaum
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy