(obsolete) How do I safely upgrade a customized Foswiki installation

Note: this topic is obsolete, as Foswiki since versions 1.1.x have a proper upgrade system. Topic kept for historical purposes only. DO NOT USE.

For people having customized their Foswiki installation, or just cautious people that want to upgrade, I have made a (linux/unix) shell script (attached to this page) foswiki-upgrade-check to automate the process. This script will apply a Foswiki upgrade on your installation, performing only the safe copies (based on a reference directory containing the distribution on which your Foswiki installation is based, and a directory containing the new release) and prompting you for merging interactively via the tool of your choice: kdiff3(default), tkdiff, xxdiff, or meld

Download: http://translate.foswiki.org/pub/Support/HowDoIUpgradeSafelyACustomizedFoswikiInstallation/foswiki-upgrade-check foswiki-upgrade-check

Usage:

foswiki-upgrade-check [options] install_dir old_release new_release
This allows you to check what a foswiki upgrade would do, and perform it
with interactive merging. Arguments are:
  - install_dir your foswiki directory, where the current instance runs
    It can be on a remote server, see REMOTE MODE below
  - old_release a directory containing the distributed release for it
  - new_release a directory containing the distributed new version

Quickstart:
If your foswiki is currently at version 1.0.4, runs in /www/fw, and you want 
to upgrade to 1.0.5. Untar the orginal 1.0.4 and 1.0.5 distribs in /tmp, and
run:
# [1] to see what may need manual merge, run:
# (it will also check that the file permissions allow the upgrade)
foswiki-upgrade-check /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5
# [2a] then, to perform the upgrade on the same server, run:
foswiki-upgrade-check -y /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5
# [2b] or, if fowsiki runs in /www/fw on host "server" as user www-data, run:
foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5
# [3] In your web browser, go to the .../bin/configure admin url, and save
the config to perform config iupgrades.

Without options, it just lists the conflicting files that may require merging.
It also saves this list in the file /tmp/fuc.$USER.list for reference.
Run with -y to actually perform the upgrade. The script will try to do as much
as possible automatically, and will prompt you to interactively resolve merge
conflicts it cannot solve via external graphical merge programs.

Note that if you updated plugins since last upgrade, the script may mistake
the new version as local changes. To avoid this, copy all the files listed for
these plugins from /tmp/fuc.$USER.list into a whitelist file (e.g: /tmp/fuc.wl)
that you will use with the -wl option to force their upgrade.

It makes a backup first to allow a rollback, but it is advised to backup your 
installation yourself, just to be sure...

In the interactive merge programs, 
  - left window is the old_release
  - middle window is install_dir
  - right window is new_release
For kdiff3 (default), for each conflict choosing the version of the 
old_release is Ctrl-1, install_dir is Ctrl-2, new_release is Ctrl-3,
so you can drive the merging process entirely via the keyboard, 
e.g: Ctrl-3, Ctrl-S, Ctrl-Q
Also, be sure to run "kdedialog" to disable kdiff3 verbose debug output

If a file listed with conflicts but the interactive merge programs window does
not appear, it is because all the merges could actually be performed
automatically for this file by the external tool.

REMOTE MODE: if install_dir is a remote dir (in the form user@host:dir), 
a remote server installation is performed as in the exemple below. E.g:
  foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Fw-1.0.4 /tmp/Fw-1.0.5
In this mode, the script goes to the server, checks what files should be
ugraded, bring them back on the local machine to perform the interactive merge
conflict resolutions, and prompts for uploading the upgrade to the server.
The account "user" on the host should have write permission to files in the 
wiki installation. As connections will be done via ssh and rsync, you may want
to install your public key as authorized_keys on this account for not having
to enter your password many times. The -z option compresses transfers.

Options:
  -v      verbose: lists all operations that are done
  -a      lists all files that would be modified, not just the "to merge" ones.
  -s      does not merge, but shows the diffs for all conflicts (use kdiff3)
  -y      yes, really "do it" performs the copy & merge
          you must have write permission to the install_dir files.
          Please make a BACKUP before!!!
          If you do not want to make a full backup, you can backup the files
          listed with the -a option that will be changed by the upgrade.
          Note that if the install_dir is remote, the changes will only be 
          made on a local copy, and you will be invited to rerun the command
          with -ry to perform the actual upgrade on the remoter server.
  -Y      same as -y, but on each conflict, prompts for what to do: 
          interactive merging, keeping original, or using new file. 
          Useful to force new files for all plugins that have been updated 
          via the web interface.
  -z      compresses transfers, useful for remote mode over slow lines
  -bl FILE    "Blacklists" files in FILE (listed one per line). These files 
          will be kept untouched in case of merge conflicts
  -wl FILE    "Whitelists" files in FILE (listed one per line). These files 
          will be overriden by the one in new_release in case of merge
          conflicts. Intended use is to list conflict into a file, by running
          first without options, edit the file to keep only the lines 
          describing the files you want to force upgrade of (typically
          distributed plugins that you may have upgraded from the configure
          interface), and use this file as whitelist for -y runs.
  -T DIR  uses DIR instead of tmp to store temporary files
  -l      lists all files of the installation that will be probed for upgrade
  -kd3    uses kdiff3 to perform interactive merges (default)
  -xxd    uses xxdiff to perform interactive merges
  -tkd    uses tkdiff to perform interactive merges
  -mld    uses meld to perform interactive merges
  -mfd DIR does not merge interactively, but instead in case of conflict take
          file from DIR. Useful if DIR is a copy of your prod instance you 
          upgraded previously to test things and do not want to repeat manual
          process and copy what you did there
  -nom    no merge: for all conflicts keep your files
  -mo FILE   merges only the file FILE into a copy, and exit. Useful to check
          things, or re-do the merge for one file if you have regrets. 
          Remember to use a backup of the old install_dir for this.
  -bd DIR specifies the backup dir (default: /tmp/foswiki-backup-DATE)
  -f      forces run: does not check that dirs are actually a foswiki install,
          allowing to use the script for other things than foswiki

Do not forget to fix files permissions after an upgrade, for instance by a 
chown -R apache:apache ., chown -R www-data:www-data ., chmod -R a+rwX .
depending on your installation, and also go to the .../bin/configure admin
url, to check for errors and warnings, and save the config to perform config
iupgrades.

The "new_release" dir can also be the distributed "upgrade" package subset, but
this is not recommended. It is advised to use the full release for the new
version as otherwise small discrepencies in the page headers may appear on
susequent upgrades, forcing cumbersome manual merges for just the %TOPICINFO
line.

Temporary files are kept for reference in /tmp/fuc.$USER feel free to remove
this directory once everything is working.

v1.11 2015-04-24 -z (compression) option for remote mode
v1.10 2013-11-29 check dependencies at start. save list of conflicts
v1.9  2013-02-02 files/dirs paths in options arguments can now be relative
v1.8  2011-12-07 remote mode was buggy
v1.7  2011-11-08 new remote mode and (hidden) -ry option
                 new -mo and -bl & -wl, blacklist & whitelist options
                 better check for file and dirs permissions, and binary files
v1.6  2011-04-20 -Y option for contributions by George Clark
                 auto-merges if the conflict is only in the topicinfo line
                 checks that install_dir files are writable
v1.5  2010-09-23 -f force option to be able to use the script on any data
v1.4  2010-03-28 -a and -l options to help backuping
v1.3  2010-02-26 dir arguments can now be relative
v1.2  2009-09-22 -s option, kdiff3 merge is now automatic if no conflicts
                 detects case where file is new but different
                 avoid merging binary files, just copy over.
v1.1  2009-08-18 -nom was not working
v1.0  2009-04-26 published on http://foswiki.org/Support/HowDoIUpgradeSafelyACustomizedFoswikiInstallation

-- ColasNahaboo - 28 Mar 2010

Would it possible for you provide a pseudo-code version of the script? I'm considering re-implementing this for Windows. But the script is far from self-explanatory for a non-Unix person.

-- SeanMorgan - 17 Aug 2009

Yes, basically:

• Let's say that I is the installed wiki, O the old distribution used to install it, U the new upgrade or full distrib
• the script goes in the U dir, and (recursively) for all the files F (not dirs) it finds there, it decides what to do:
• if F does not exists in I, it copies F from U to I
• else, if F in I and U are the same do nothing, but if F in I is different,:
  • if F is the same in O and I, it means the file has changed in the new distrib but we didn't modify it locally. just silently copy F from U to I
  • else if F is the same in U and O, it means the file was not changed in the distrib but we changed it locally. we do nothing to keep our version
  • else we have a conflict. Either just print F, or if -y is given, we do a 3-way merge interactively between O, I, U versions of F and save result in I

-- ColasNahaboo - 19 Aug 2009

Thank-you for the explanation.

Related info: this tool comparison shows platform support, as well as feature support (i.e., the 3-way merge):

• http://en.wikipedia.org/wiki/Comparison_of_file_comparison_tools

With the exception of meld the tools you mentioned are all available on Windows.

-- SeanMorgan - 31 Aug 2009

It would be helpful for some additional tips/solutions for those without a graphical environment (which all of the above seem predicated on?) Are there command line diff tools that are compatible? The use case here would be a minimal VM instance, on debian or other.

-- CraigBowers - 01 Sep 2009

The "sdiff" tool that comes with the gnu diff utilities can do a side-by-side text based merge. For an example, the gentoo package upgrade tools (dispatch-conf) uses sdiff to interactively merge configuration files that are modified after each package upgrade. sdiff works pretty well. The "dispatch-conf" tool presents each "protected" file that changes with a simple diff. Among other options, you can choose to keep the new version, temporarily skip the new version and keep the old one, delete the new version, or do a side-by-side merge. With the merge, you can keep the left or right version of blocks of changed lines.

-- GeorgeClark - 02 Sep 2009

"sdiff" is not a 3-way merge. Just 2 ways. For headless servers, what I do is:

• run foswiki-upgrade-check on the server to list the conflicting files. take note of them for instance by foswiki-upgrade-check >/tmp/fwu.list
• copy the server install on your local workstation (by rsync -aHSx --delete --force SERVER:SRC/ DEST/ )
• do the foswiki-upgrade-check locally (in DEST/) with your graphical merge tool
• either re-upload your upgraded local copy to another place on the server (let's say /tmp/fwu ), or better just the resolved conflicts ( rsync -aHSx --delete --force -R `cat /tmp/fwu.list` SERVER/tmp/fwu/ ) if you copied /tmp/fwu.list from the server locally
• on the server, perform the automatic merge with foswiki-upgrade-check -y -mfd /tmp/fwu

uploaded v1.2 of the script

-- ColasNahaboo - 22 Sep 2009

uploaded v1.3 2010-02-26 dir arguments can now be relative

-- ColasNahaboo - 26 Feb 2010

uploaded v1.4 2010-03-28 -a and -l options to help backing up

NOTE it is recommended that you do not use the "Foswiki-upgrade-*" packages, as they omit some files that have actually changed (in minor ways like the hidden TOPICINFO metadata), it will generate a lot of conflict merges at next update that will not be automatically solvable.

-- ColasNahaboo - 28 Mar 2010

With Foswiki-1.1, there are a large number of .gz files that are not detected as binary. .gz needs to be added as a binary extension. Also I've found that most of the time - especially with all the extensions now shipped by default, that I just want to use the new version of the files - all the JQuery files, for example.

I've got a new version that I've attached that resolves the .gz issue, and adds a prompt in the shell. This allows the graphical merge to be skipped most of the time. Answer with 1, 2 or 3.

How do you want to handle conflict: /home/user/foswiki/Foswiki-1.1.0-RC1/data/Sandbox/WebNotify.txt ?
1) Merge files
2) Keep original
3) Use new file
#? 

-- GeorgeClark - 30 Sep 2010

Thanks Georges, I have incorporated your changes. The prompting for conflict is now enabled if -Y is given instead of -y, and I have changed it to stay on a line, and to default to interactive merge on a simple return:

Merging data/System/PatternSkin.txt
  [Enter, 1, m] = Merge, [2, o] = keep Original, [3, n] = use New file: 2
  Keeping original

New version v1.6 2011-04-20 uploaded as attachment to this topic.

-- ColasNahaboo - 20 Apr 2011

New version v1.7 2011-11-08 uploaded as attachment to this topic. Main new feature: remote mode to upgrade a remote server, generates files to perform a rollback.

-- ColasNahaboo - 09 Nov 2011

New version v1.8 2011-12-07 uploaded as attachment to this topic. Bugfix to 1.7 that was always using update files in some cases. please upgrade

-- ColasNahaboo - 07 Dec 2011

Version v1.10 2013-11-29 uploaded as attachment to this topic.

• remote mode fixed
• files/dirs paths in options arguments can now be relative
• checks for the availability of the chosen non-standard commands (kdiff3, tkdiff, xxdiff, meld) before starting
• the list of conflict printed on first run is now also stored in a file, to ease building whitelists & blacklists for the -wl & -bl options

-- ColasNahaboo - 29 Nov 2013

Version v1.11 2015-04-24 uploaded as attachment to this topic.

• new -z option to compress transfers for remote upgrades
• scripts checks that install_dir and old_release are actually the same version of foswiki, for a proper 3-way merge

-- ColasNahaboo - 28 Apr 2015

I marked this topic as obsolete, as:

• Foswiki is now extremely clean, and upgrading minor versions via the upgrade distribution works very well
• In the rare event where you have to patch Foswiki code, I recommend packaging your changes as patches and re-apply them on each upgrade. This will be more controllable than a general "blind" merge like this script does
• Merging everything like this script does runs the risk of your installation become more and more different than the standard

So my recommendations are:

• If you run foswiki 1.x.x, just install a fresh Foswiki 2.x, create all the needed webs, and copy over your data, that is your web topics + attachments less the topics defined in the data/_default/ (but copy also your WebHomes) so you start afresh. The tweak your data if you see incompatibilities (macro use, missing plugins, ...)
• Then upgrade with the recommended way: just unpacking the "upgrade" dists for minor upgrades.

-- ColasNahaboo - 21 Nov 2015