BuildContrib Cookbook
This topic has been partially updated for using git, but needs validation and further updates.
This is a cookbook of recipes for using
BuildContrib.
BuildContrib is a powerful, multi-faceted build system, used to build Foswiki extensions and the Foswiki core itself. It started out as a simple perl clone of ANT, but has gained a diverse range of other functionality since then.
Extensions should be checked into git and pushed to github, so that's assumed here. Anyone who develops outside git is asking for trouble, and is pretty much on their own. See
HowToStartExtensionDevelopmentInGit for help getting started with it.
Insert the name and type of your module below, and hit "rewrite instructions" to rewrite the instructions for your module. See the
Extensions web for a description of the different types of extension module. However note that AddOn is no longer used and should be avoided.
Recipe 1: Setting up your dev environment
The best way to develop is in a cloned git repository. See
GitRepository for help in checking one out. (Subversion has been frozen and should not be used.) Once you have a checkout area, follow the installation instructions to
configure it as a running Foswiki so you can test your code in the browser. Then set these environment variables (adjust for your shell). The 2nd one is a path with two directories. The last one will pick up the recommended version of
perltidy
export FOSWIKI_HOME=%GIT%/core
export FOSWIKI_LIBS=$FOSWIKI_HOME/lib:$FOSWIKI_HOME/lib/CPAN/lib
export PERL5LIB=%GIT%/BuildContrib/lib
The beauty of using a checkout area is that if you make a mistake, it's easy to revert your changes, and you are always testing against the latest code. By installing the "distro" github repository, you already have the pieces needed for development.
Use
pseudo-install.pl
to install all the developer components into the core, thus.
cd /var/www/foswiki/core && ./pseudo-install.pl developer
If you intend to create a "Contrib" or "Skin" you'll also need to install the EmptyContrib
cd /var/www/foswiki/core && ./pseudo-install.pl EmptyContrib
If you want to be able to check your work into github directly, you need to be authorized. See
GettingStarted
Recipe 2: Creating a new extension
This recipe describes the process of building a new extension called ElectricPlugin, using the
create_new_extension.pl
script installed with
BuildContrib. (Before you go any further. If you intend to create a Contrib or Skin,
pseudo-install EmptyContrib )
-
cd /var/www/foswiki
-
perl core/create_new_extension.pl ElectricPlugin
This will create the ElectricPlugin file hierarchy, including
build.pl
,
DEPENDENCIES
and
MANIFEST
. At this point, the Extension Topic (and all other files) are copied from Empty(Plugin/Contrib), and you will need to change those to reflect what you are working on.
Recipe 3: Making MANIFEST and DEPENDENCIES files
The
MANIFEST
file lists files that are
released with the extension. Not all files in the source tree will be released; for example, it is unusual to release developer tests and support scripts such as those used for benchmarking.
The
DEPENDENCIES
file lists external dependencies, such as those on CPAN modules, or other Foswiki modules. The format of the
DEPENDENCIES
file is one dependency per line;
dependency
,
version
,
type
, and
comment
are comma-separated fields specified per dependency, defined below.
Storable,>=0,cpan,Required for saving attachment hash values and parameters
htmldoc,,external,Required
- Dependency name Package or module name to be installed. For Foswiki modules it is a fully qualified name, for example
Foswiki::Plugins::SafeWikiPlugin
, For CPAN modules, this should be the name as installed from the CPAN shell. For external packages, and for best compatibility with the .deb
Debian packager, the name should have a unique hit when searched with apt-cache search <name>
- Version If a minimum version of the package is required, this should be a simple expression
>10.1
for example
- Type This is one of
perl
, cpan
or external
. perl
dependencies are only used for other Foswiki modules. Note that the Foswiki installer will not install external packages, however if the dependency matches a debian package, it will be added to the .deb
package as a dependency.
- Description Description of the dependency. If the description starts with the word
Optional
, it will never be automatically installed.
After you run
create_new_extension.pl
the initial
MANIFEST
and
DEPENDENCIES
files are empty. You can generate draft contents thus:
-
cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
-
perl build.pl manifest
-
perl build.pl dependencies
(needs the CPAN module B::PerlReq
)
These targets "guess" the content of the files; you need to apply some intelligence to derive the actual content.
Caution The dependencies
build target assumes that all dependencies are already installed on the system. The tool will not detect unresolved dependencies that cause compile errors in the modules.
You can run these targets whenever you want, even after you have generated what you believe are correct
MANIFEST
and
DEPENDENCIES
files.
Recipe 4: Making unit tests
We strongly recommend using test-first development, and ensuring that you have unit tests in place for all your extensions. Unfortunately many people are too lazy, or too stupid (like the author of BuildContrib), to write tests for their extensions. If you are one of the wise:
-
mkdir -p /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin
- Create a testcase. The easiest way to do this is to base it on an existing testcase. For an example, see
/var/www/foswiki/CommentPlugin/test/unit/CommentPlugin
. You can find the complete library of unit tests for core and all pseudo-installed extensions in /var/www/foswiki/core/test/unit
Recipe 4b: Running the unit tests
- do not ignore test failures - maintaining a quality test suite is a major help to yourself and others in maintaining extensions
There are two ways to run the tests. You can run them from within the extension directory, or run them from within core. Running from within core simplifies some things.
-
cd /var/www/foswiki/core/test/unit
-
perl ../bin/TestRunner.pl ElectricPluginSuite
(or if you didn't create the suite)
-
perl ../bin/TestRunner.pl ElectricPluginTests
Alternative:
- To run the tests,
cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
and perl build.pl test
This method will only run Test "Suite" modules.
-
build.pl
will print the command line used to run the tests, so you can easily copy-paste-modify the command line to home in on a failing test
Recipe 5: Debugging unit tests
Sometimes a unit test will fail (that's why you have them, after all) and you will need to debug. This is not so easily achieved with
build.pl
since it fires off separate processes to do the hard work. Fortunately there are some simple tactics you can use to help debug a test. Say that you want to know why, in your ElectricPlugin,
test_Something
reports a failure though it is supposed to work.
Recipe 5a: Isolate the failing test
At the bottom of the test run is a short TML report:
Module Failure summary
FoswikirefsPluginTests has 1 unexpected results (of 2):
- F: FoswikirefsPluginTests::test_GITREF
Here is where running the tests from within core is helpful. (The
build.pl test
cannot run individual tests.) Copy/paste the failing test line to run it again. (Note always include -clean when re-running tests).
-
cd /var/www/foswiki/core/test/unit
-
../bin/TestRunner.pl -clean FoswikirefsPluginTests::test_GITREF
From here you can add print statements, or make other changes to isolate the issue.
Recipe 5b: Use a debugger
Needs updating
The following recipe will guide you to a Perl debugger session which stops at the first line of
sub test_Something
. It is less scary than it looks at first glance, you only need to type in what's in
boldface
.
-
$
cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
-
$
export PERL5OPT=-d
- ...or the equivalent for your shell. This environment variable contains options which well be passed to all invocations of perl from now on.
-
$
perl build.pl test
-
Loading DB routines from perl5db.pl version 1.28
(...)
main::(build.pl:48): $build = new BuildBuild();
DB<1>
c
- Just continue, we are not going to find bugs in
build.pl
(we hope!)
-
Running tests in /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin/TemplateToolkitPluginSuite.pm
(...)
main::(/var/www/foswiki/test/bin/TestRunner.pl:3):
3: require 5.006;
DB<1>
b postpone ElectricPluginTests::test_Something
- Daughter DB session, maybe in an new xterm. At the current line, your test suite has not yet been compiled, that/home/foswiki/core/test/bin/TestRunner.pl /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm's what
postpone
is for.
-
DB<2>
c
- Start
TestRunner.pl
. Some test cases may run before test_Something
(the ordering is hardly predictable), but eventually...
-
ElectricPluginTests::test_Something(ElectricPluginTests.pm:57):
57: my $this = shift;
DB<3>
- Eventually, you're where you can start tracing. Phew! After you're done, you have to quit (
q
) twice, for both parent and daughter session.
Final hint: If
all your test cases fail, but perl never stops at the break points you set with this recipe, set a breakpoint at the init routine:
b postpone ElectricPluginTests::set_up
Recipe 6: Testing your release package
You build and test your release package as follows.
- If you followed recipe 5, remember to
./pseudo-install.pl -uninstall ElectricPlugin
first, or bad things will happen
-
cd /var/www/foswiki
-
perl ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin/build.pl release
- Copy package files into Foswiki root
cp ElectricPlugin/ElectricPlugin* core/
-
cd core/
-
perl ElectricPlugin_installer -r install
- Test in the browser, check the documentation etc etc
Recipe 7: Uploading your work
Please don't skip Recipe 6! When an extension is uploaded, it becomes visible to the configure "Find Extensions" interface. Extensions that Foswiki advertises in the installer should at least install and run without errors using the Plugin topic. Extensions that don't cleanly install or function reflect badly on overall Foswiki quality. Note: If your extension will not work or is not tested in some environments, note this in the package form. And if it has not been thoroughly tested, or has known issues, please upload to the
Extensions Testing Web.
The best way to upload your release package is to use the
upload
target.
cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
and then
perl build.pl upload
It will offer to upload to foswiki.org (you can change this to upload to another repository, for example if you are using BuildContrib to build company-specific plugins). Use your Foswiki username and password.
It will cache them to a local, unencrypted file,
$HOME/.buildcontrib
! If you can't use the
upload
target, then manually upload/attach the files that were built when you did
perl build.pl release
.
If your upload is large, or your upload link is very slow and takes longer than 3 minutes, it may fail with a 500 Internal Server Error
. This is because of the 3 minute timeout on the fcgi process. Change the upload to use plain CGI by setting the Scripts path to http://foswiki.org/cgi
Note that the
perl build.pl upload
command will attempt to upload any attachments in the metadata of the Plugin topic. If your plugin has attached any files that you don't intend to upload, delete the metadata or the upload will fail with missing file errors.
--
Contributors: CrawfordCurrie,
SvenDowideit,
TWiki:Main.HaraldJoerg,
GeorgeClark,
NicolasWrobel
Discussion
In
Tasks.Item8531, we finally stopped checking in .gz and compressed versions of js and css files, which is a great step forward.
But now, autogenerated MANIFESTs are much less useful than they used to be, and large plugins like
TinyMCEPlugin,
JQueryPlugin, anything with many .css and .js files is very tedious to confidently come up with a comprehensive MANIFEST.
So below are a couple of
vim
tricks that I finally got working, so that I can reliably rebuild the manifest automatically and apply the following regexes to get the minified/gzipped versions where appropriate
- Assume extension directory has all minified and gzipped files removed
- Generate a new MANIFEST using
./build.pl manifest
- Insert .gz entries for any .css or .js file that does NOT have .uncompressed in their name:
%s/^\(.*uncompressed.*\)\@!\(.*\)\.\(css\|js\)\(.*$\)/\1\2.\3\r\1\2.\3.gz/
- Insert .gz AND minified entries for all .uncompressed.files:
%s/^\(.*\)\.\(uncompressed\)\.\(js\|css\)\(.*\)$/\1.\2.\3\4\r\1.\3\4\r\1.\3.gz\4
Added a full recipe for updating
TinyMCEPlugin at
Development.TinyMCEPlugin
Probably, this is a sign that the gzipping/minificaiton work in
BuildContrib is incomplete
--
PaulHarvey - 17 Mar 2010
You might want to have a look at
depgen, which I recently developed to generate DEPENDENCIES, the related challenge with large plugins. It actually scans sources, so I suppose it could be adapted to find css/js references if you aren't too cute about how they are generated. In any case, it does solve the DEPENDENCIES challenge. (The off-line tasks package has 50 dependencies scattered across about 140 source files.) Run it --man for docs (but not as
root
).
Of course it would be great if build.pl handled all this correctly - I couldn't figure out how to get the TWiki version to do the right things, especially for a complex object. Thus this tool.
Enjoy.
--
TimotheLitt - 08 Oct 2012
build.pl
has been able to do this for a long time (perhaps even pre-fork?);
perl build.pl dependencies
Again it doesn't handle JS or CSS dependenices; that's really tricky to do.
--
CrawfordCurrie - 08 Oct 2012