Feature Proposal: Configure should install commonly tailored topics so that new releases don't overlay customized topics.

Motivation

We've needed a way to avoid overlaying tailored topics in new releases. Typically this gets done by building separate release and upgrade distribution files.

Description and Documentation

Instead of using multiple versions of the release archives, store the "normally removed" files in the in a subweb of System. 404 handler will "restore" the files only if they are missing from the live system. This should only happen on initial installation.

See also Item780

Examples

Add a new System subweb, System/Distribution - containing the topics that would normally be tailored by a local installation. This subweb would contain copies the files that would normally be removed from an "upgrade" distribution, and remove them from their normal locations

• System/Distribution.Main_SitePreference
• System/Distribution.Main_WebHome.txt
• System/Distribution.Main_WebPreferences.txt
• System/Distribution.Sandbox_WebHome.txt
• System/Distribution.Sandbox_WebPreferences.txt
• System/Distribution.System_WebHome.txt
• System/Distribution.System_WebPreferences.txt
• System/Distribution.Trash_WebHome.txt
• System/Distribution.Trash_WebPreferences.txt
• System/Distribution.Trash_TrashAttachments.txt
• System/Distribution._default_WebHome.txt
• System/Distribution._default_WebPreferences.txt
• System/Distribution._default_WebLeftBar.txt
• System/Distribution._empty_WebPreferences.txt
• System/Distribution.TWiki_WebPreferences.txt

Unsure if any of these are required

• System/InterWikis.txt May not be necessary due to new setting for a local topic
• System/NewUserTemplate.txt
• System/UserRegistration.txt

bin/view in 404 handling will attempt to "restore" any of these missing topics. and bin/configure could implement a one-time warning, maybe based upon the date of the topic changes, so that an admin knows that these topics have changed and might need to be updated.

If we ever ship a default "empty the trash" script, it could also use the Trash_TrashAttachment to restore that topic to it's original state. (Or if has been accidentally removed, revisiting configure will restore the topic).

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

Map each file to the correct destination using Configure::Func::mapTarget to accommodate renamed Webs or Topics.

• Should we consider shipping a list of previous topic MD5 hash values. If the topic exists but matches a previously shipped MD5 hash, then it should still be acceptable to copy in the new file. Not necessary.
• One-time-only (save an expert setting?) warn the user if the new files don't match the existing files so they can check for changes
• Do we need to accommodate anything in pub. (I don't believe so with the above list of topics).

-- Contributors: GeorgeClark - 30 Aug 2010

Discussion

Excellent idea George. However, I'm wondering if this overlaps with the "phantom topics" idea that's been thrown around on IRC. Basically, a "phantom" topic is one that doesn't physically exist in the requested web, but exists elsewhere, like data/_tailored/Main or data/_phantom/Main. Then somehow, we dream up a way to map the missing (requested) topic onto the phantom location.

This would also mean we wouldn't have to ship the Web* topics.

To some extent MichaelDaum's AutoTemplatePlugin achieves this using topic name rules.

I would like to think that we could invent a topicMissing() plugin event handler to do the same; allow a plugin (or Foswiki core) to handle a missing topic in a configurable way.

-- PaulHarvey - 30 Aug 2010

I've added a concern to make sure its clear that I don't think we should be doing this in 1.1 - its a 1.2 thing at this late date.

additionally, there are several existing proposals for this exact same issue that we've already refused for 1.1 - including my INCLUDE multiple topics with fallback like SKIN path, the rejected AutoView solution that Micha proposed, and another again based on tmpl's.

to my mind, we've wandered back into the territory where the overlap between INCLUDE using topics and SKIN path using topics and tmpl's shows that one should be deprecated...

back to your proposal - I'm not sure your approach needs a new web (though it might just make things really obvious ) - instead, if we accept that the SYSTEMWEB.Default* (or if you use a separate default web DEFAULT.*) is used for any topic that can be over-ridden both for the entire system (ie, in MAINWEB), or for a particular web, we can either mix them in magically in the Store code (I've done this previously in TagsPlugin), or provide (as you suggest) a trivial customise_me topic that has 2 bits to it - an INCLUDE{DEFAULT.TOPIC} and a customise me button which then copies the DEFAULT.

the idea here is that any un-customised topics are always magically updated due to the INCLUDE, and we can even create a topic template for the admin to show differences if the shipped version has updated the DEFAULT topic after the customised version was made.

TODO : need to work out how to do the same wrt webs that where created using an extensions' custom _web and that sort of thing.

I'm not that thrilled about using configure for this kind of thing - people do still unzip into their foswiki dir, and tbh, until configure can be run from the cmd line, its too big an ask.

-- SvenDowideit - 30 Aug 2010

So my proposal was intended to be very simple, and based upon a config checker. If we choose to still ship the files in their ultimate locations, then the checker function becomes a way to restore back a small number of critical files if they are accidentally deleted. If omitted from the archive, then configure takes care of installing them if it's a new installation.

If the admin has deleted the file and really doesn't want us to restore it, then delete it from the setaside location as well and checker doesn't touch it. The checker is driven by the contents of the setaside location. So if we don't put anything there in the release package, it will do nothing.

I didn't want to boil the ocean here or solve the larger issue of generic tailored topics. I suspect that this could be a somewhat useful checker function even if we don't omit the topics. It also "saves" an install if a user accidentally installs an upgrade package as a new installation.

I don't know how to alleviate your objections wrt/ 1.1 timeframe. It was intended to be lightweight enough for 1.1 and if it's delayed, then it could still be useful as recovery mechanism for a damaged installation. But any urgency is gone.

For the unzip / don't run configure installation crowd, we could add a small perl script to do the copy .. or include an additional zipfile in the distribution that will expand just the omitted files into their correct locations. Or build a version that includes all files. which is what we do today anyway.

-- GeorgeClark - 30 Aug 2010

I think the goal here is excellent, and if we can accomplish easier upgrades it could make a huge difference for Foswiki administrators, and Foswiki adoption in general.

Some goals to aim for:

• Never over-write a customized topic (any customized topic) on upgrade
• Make it easy for administrators to see "conflicts" between upgraded default topics, and customized default topics, and resolve those conflicts.

I like the idea of using some sort of phantom topics or default topics web - or perhaps some sort of "conflict topics" web, which would be a place to put the topics that conflict - possibly with naming and visibility similar to Trash web (and an ability to "restore" the default topic, similar to a deleted topic).

Perhaps consider an upgrade script that simply checks who most recently edited the topic? If it was ProjectContributor or AdminGroup, the topic is safe to upgrade with no conflict. Anyone else, and we should generate a conflict topic and let the administrator decide how to handle it?

-- LeilaPearson - 12 Sep 2010

I like this. It needs to handle pub too.

How do you distinguish between a topic that has been locally tailored and one that was planted by configure in the last release (but now has a new revision)?

-- CrawfordCurrie - 07 Dec 2010

I think that this is only one part of the solution. The proposal UpgradeSafeCustomisationsOfSystemTopics and MakeUserRegistrationCustomizable are both needed. We have some topics that really don't matter when modified in a new release. WebHome, WebNotify, .. Typically changes to them are not really operational and don't need to be picked up by an existing site. We don't need the overhead of INCLUDE. There are other topics where the contents is operational. Such as Search, and Registration. An INCLUDE or Template approach is better here. (or more likely both apply).

Maybe we could add a checker that reports "The following topics differ from their distribution, check this checkbox to verify you've reviewed the changes.

I've got to run off now, but I'll add some "upgrade flows" later to look at how this would work for various topics.

-- GeorgeClark - 07 Dec 2010

Before you guys fix too much remember that UserRegistration topic already has been fixed so a tailoring is not overwritten by upgrades any more.

ChangePassword and ResetPassword and ChangeEmailAddress are now tailored by setting in SitePreferences and should not need tailoring any longer.

I by far prefer eliminating the need for altering the distributed topics instead of trying to create geek solutions. How will the admin know to merge his tailored System topic with the update we have made? Maybe a feature we added requires that the topic is totally updated. Maybe we plug a security hole. We just replace one problem by another problem.

I really warn against such a solution.

Rather think about further eliminating the need to alter System web topics.

-- KennethLavrsen - 07 Dec 2010

If we can reach the point that an unzip of Foswiki on top of an existing installation doesn't break local modifications, then this proposal is moot. As long as you have to build two installer packages - one for existing, one for upgrades, then this proposal still has some (questionable?) merit. My objective was to build in the process you go through with each release, so you only ever build an "upgrade" package. If the end result of other work is that there is no difference between upgrade.tgz and foswiki.tgz, then flush this. However I suspect that WebHome and WebPreferences for Main and Sandbox will always have to be deleted from your Upgrade package.

You've raised "How will admin know to merge..." How do they know today if you've deleted the topic from the Upgrade package. It's not there to even compare to see if something changed.

If it's a topic you delete from Upgrade.tgz, then we move it to another location, let configure, or maybe even the first execution of view copy in the missing topic. And you don't have to prepare two packages. Users can't install the wrong one. If the architecture changes so you don't have to delete the topic, then it gets removed from this process. If end result of other tailoring is we never have to build an upgrade package, then this goes away.

If this doesn't make sense, then flip the status to Rejected by consensus. It's not all that major, and only impacts building the packages.

-- GeorgeClark - 09 Dec 2010

I don't think upgrade packages make this proposal moot. The manual process increases the risk, and if you accidentally override a topic you have customized then you might loose all your work. I would rather that customized topics are stored separately from shipped topics, such as another web, but I don't know the best way to do this.

So I still think a proposal to make upgrades easier is needed. But unfortunately I don't think we are any closer to finding a solution that is acceptable to everyone.

-- AndrewJones - 09 Dec 2010

IRC Discussion. Maybe getting a bit closer.

(06:05:14 PM) gac410: Lavr:  A couple of more comments on Foswiki:Development/ShipCommonlyTailoredTopicsInConfigure   And close it out.  I'm okay to change to rejected and move on if you don't see the value.
(06:05:16 PM) foswikibot: http://foswiki.org/Development/ShipCommonlyTailoredTopicsInConfigure
(06:06:40 PM) gac410: I see it only covering the small number of topics that will probably *always* be deleted from an upgrade package.
(06:08:51 PM) Lavr: My ultimate goal is for sure to have the normal package being an upg pack you can just throw on top
(06:09:17 PM) Lavr: Not there yet but for each release we are getting closer
(06:10:00 PM) Lavr: Currently I remove WebPreferences UNLESS there was a security problem with it
(06:10:20 PM) gac410: Maybe I'm thick but I can't see any way handle Main/WebPreferences or Sandbox/WebPreferences   (and WebHome)  without providing them for new installs.  
(06:10:54 PM) gac410: It's not a web without WebPreferences so we must put something there.    
(06:10:59 PM) Lavr: No  I do not yet have a good solution to WebPreferences
(06:12:43 PM) Lavr: I have been thinking about having a default prefs scheme so if you define a UserWebPreferences or LocalWebPreferences it takes precidence (spelling?)
(06:12:56 PM) gac410: System is close to covered.  The various include / template type stuff works well      But WebHome  and WebPreferences.    Those are going to be tough I suspect.
(06:13:28 PM) gac410: That will need some other work then -    A web is by definition a directory with a WebPreferences.txt   file
(06:13:53 PM) gac410: If it becomes optional, then is an empty directory a web?  
(06:14:08 PM) Lavr: Yes. but all the webs we ship could have a WebPreferences like now
(06:14:30 PM) gac410: Right -  but if we ship it named WebPreferences,  then you need to remove it from an upgrade.
(06:14:42 PM) Lavr: But if you create a LocalWebPreferences it would take over
(06:15:13 PM) wbniv: and you would expose directories under data/ that were not previously public
(06:15:20 PM) gac410: Ah.   Like SitePreferences vs. DefaultPreferences.    Though we do ship a SitePreferences as well. 
(06:15:21 PM) wbniv: because they didn't have a WebPreferences.txt
(06:15:31 PM) Lavr: Idea being that you would use this in Main web, System web, and Sandbox only
(06:16:06 PM) Lavr: It is one idea and there could be other ways
(06:16:06 PM) gac410: Have to consider perf. impact -  It adds another topic to the Prefs search hierarchy.   Which is bad already.
(06:17:06 PM) Lavr: It is the performance concern that has made me not proposed it
(06:19:13 PM) Lavr: System.WebHome is not a normal tailoring object. It is mainly Main and Sandbox.WebHome that people want preserved
(06:19:55 PM) Lavr: And for those a merge with the new version wheb updating would still be unwanted
(06:20:05 PM) gac410: Agreed.     And our changes are *typically* of small consequence on them.   
(06:21:10 PM) gac410: The Prefs are dangerous because unlike Registration, Login, etc.   They are read on *every* hit.     So adding a shadow / default version is signinficant.
(06:22:32 PM) Lavr: For WebHome and maybe also WebPreferences the solution could be to distribute the webs with WebPreferencesDefault and WebHomeDefault in the 3 webs
(06:22:45 PM) Lavr: And no WebHome or WebPreferences
(06:23:38 PM) Lavr: And when you rub configure first time it would copy the defaults to the normal if the normals are not present
(06:23:52 PM) Lavr: Like we also create WikiUsers
(06:24:29 PM) Lavr: .. when first user registers
(06:24:48 PM) gac410: That's exactly what I was proposing ...  except configure stored the copies in it's own space.
(06:25:44 PM) gac410: or make it handled in the 404 process.  for the sites that don't run configure.  
(06:25:59 PM) Lavr: If it is only WebHome and WebPreferences in the 3 webs a special space would not really be needed
(06:26:35 PM) Lavr: Yes. Creating them first time you run Foswiki could also be a way
(06:28:02 PM) gac410: Putting them in the web might be confusing.   Users editing Default and then wondering why it doesn't work.
(06:28:36 PM) Lavr: That is how we handle WebLeftBar etc now. Goes OK
(06:29:36 PM) gac410: We also have to consider tailored _Default and _Empty webs.  
(06:30:14 PM) Lavr: And then when you upgrade you have a place where you can see what new things we have added
(06:31:13 PM) Lavr: _Empty is probably always empty. The _default you may also tailor yes
(06:31:45 PM) gac410: My leaninng was toward a _Distributed   web so they can be easily referenced by an admin without cluttering the webs for users.
(06:32:33 PM) Lavr: Without any really good science reason I feel bad about distributing more webs
(06:33:12 PM) gac410: Although starting with _ underscore it's somewhat masked from mortal users.
(06:33:44 PM) Lavr: Except you will see it as a template web in the create web UI
(06:33:53 PM) gac410: crap
(06:34:08 PM) Lavr: That we could program our way out of. just exclude that webname
(06:35:55 PM) Lavr: I would somehow feel better if these things were stored in the System web using some naming scheme. putting then in some other directory makes it difficult for devs to review the rendered versions when editing
(06:37:02 PM) gac410: A "Distributed"  sub-web  containing   MainWebHome SandboxWebHome   MainWebPreference, ... and so on?
(06:37:23 PM) Lavr: Best would be if we could upgrade from configure and let it make decisions of what not to upgrade
(06:37:59 PM) Lavr: Configure should be able to upgrade in two ways
(06:38:12 PM) gac410: The package installer is probably really close.   Any topic/attachment found with an existing ",v" file will be "checked in"   
(06:38:16 PM) Lavr: 1 download and install like Firefox does
(06:38:40 PM) gac410: Will did a trial upgrade package for one of the 1.1.1 bugs IIRC.
(06:39:03 PM) Lavr: 2. Upload a full zip (manually downloaded) to configure in an upgrade interface
(06:39:19 PM) Lavr: Like you would upgrade a router or a NAS
(06:40:01 PM) gac410: Package needs an _installer file conntaining a manifest.    It will unzip and copy each file,   mapping to renamed Webs / Topics,   and Checking-in where needed.
(06:40:38 PM) gac410: So if user renamed Sandbox to Playground,  and WebHome to HomePage ...    Package.pm will still work.
(06:41:00 PM) gac410: Provided the Foswik::cfg variables are set correctly.
(06:41:13 PM) Lavr: I would not want System topics upreved all the time. That is how TWiki used to do it in the old days and it never really added value
(06:41:51 PM) gac410: If there is no ,v file found,  then it overlays.    Only if a ,v,.  which says someone has made a local change.
(06:42:04 PM) gac410: (and manifest must flag as !noci)
(06:42:12 PM) Lavr: But it is just a matter of being able to define which should be checked in and which should be replaced
(06:42:28 PM) Lavr: ah , that could fly then
(06:43:06 PM) gac410: Have to think a bit more on what happens if a Topic does not have ,v,  but an attachment does.  But I don't think that is actually possible 
(06:43:45 PM) Lavr: Even in TWikinwe avoided ,v files for attachments.
(06:43:48 PM) gac410: And we need to have a way to "turn off" uninstall.     Accidentally uninstalling a core package might be a bad idea  ;-)
(06:43:52 PM) Lavr: In the distro I mean
(06:44:19 PM) gac410: Though a "restore"  might be better to have.    
(06:44:24 PM) Lavr: Yes. No uninstall of core :-)

-- GeorgeClark - 9 Dec 2010

I've updated the proposal to incorporate some of the points in the above discussion.

-- GeorgeClark - 19 Dec 2010

Comparing the 1.1.2 upgrade and full packages, the following files were omitted from the upgrade. Should this whole list be covered by the proposal?

< Foswiki-1.1.2/data/_default/WebHome.txt
< Foswiki-1.1.2/data/_default/WebLeftBar.txt
< Foswiki-1.1.2/data/_default/WebPreferences.txt
< Foswiki-1.1.2/data/TWiki/WebPreferences.txt
< Foswiki-1.1.2/data/Main/WikiGuest.txt
< Foswiki-1.1.2/data/Main/WebRss.txt
< Foswiki-1.1.2/data/Main/SitePreferences.txt
< Foswiki-1.1.2/data/Main/WebSearchAdvanced.txt
< Foswiki-1.1.2/data/Main/WebChanges.txt
< Foswiki-1.1.2/data/Main/WebCreateNewTopic.txt
< Foswiki-1.1.2/data/Main/WebSearch.txt
< Foswiki-1.1.2/data/Main/AdminGroup.txt
< Foswiki-1.1.2/data/Main/WebHome.txt
< Foswiki-1.1.2/data/Main/UnknownUser.txt
< Foswiki-1.1.2/data/Main/WebNotify.txt
< Foswiki-1.1.2/data/Main/WebTopicList.txt
< Foswiki-1.1.2/data/Main/WebStatistics.txt
< Foswiki-1.1.2/data/Main/UserListByDateJoined.txt
< Foswiki-1.1.2/data/Main/WebPreferences.txt
< Foswiki-1.1.2/data/Main/UserListByLocation.txt
< Foswiki-1.1.2/data/Main/WebLeftBarExample.txt
< Foswiki-1.1.2/data/Main/RegistrationAgent.txt
< Foswiki-1.1.2/data/Main/PatternSkinUserViewTemplate.txt
< Foswiki-1.1.2/data/Main/AdminUser.txt
< Foswiki-1.1.2/data/Main/WebIndex.txt
< Foswiki-1.1.2/data/Main/ProjectContributor.txt
< Foswiki-1.1.2/data/Main/UserHomepageHeader.txt
< Foswiki-1.1.2/data/Main/UserList.txt
< Foswiki-1.1.2/data/Main/UserListHeader.txt
< Foswiki-1.1.2/data/Main/WebAtom.txt
< Foswiki-1.1.2/data/Main/NobodyGroup.txt
< Foswiki-1.1.2/data/Sandbox/WebRss.txt
< Foswiki-1.1.2/data/Sandbox/WebSearchAdvanced.txt
< Foswiki-1.1.2/data/Sandbox/CommentPluginExamples.txt
< Foswiki-1.1.2/data/Sandbox/WebChanges.txt
< Foswiki-1.1.2/data/Sandbox/WebCreateNewTopic.txt
< Foswiki-1.1.2/data/Sandbox/WebSearch.txt
< Foswiki-1.1.2/data/Sandbox/CommentPluginTemplateExample.txt
< Foswiki-1.1.2/data/Sandbox/WebHome.txt
< Foswiki-1.1.2/data/Sandbox/WebNotify.txt
< Foswiki-1.1.2/data/Sandbox/PluginTestEmptyPlugin.txt
< Foswiki-1.1.2/data/Sandbox/WebTopicList.txt
< Foswiki-1.1.2/data/Sandbox/CompareRevisionsAddOnDemoTopic.txt
< Foswiki-1.1.2/data/Sandbox/WebStatistics.txt
< Foswiki-1.1.2/data/Sandbox/WebPreferences.txt
< Foswiki-1.1.2/data/Sandbox/CommentPluginExampleComments.txt
< Foswiki-1.1.2/data/Sandbox/CompareRevisionsAddOnDemoTopic.txt,v
< Foswiki-1.1.2/data/Sandbox/WebLeftBarExample.txt
< Foswiki-1.1.2/data/Sandbox/WebIndex.txt
< Foswiki-1.1.2/data/Sandbox/WebAtom.txt
< Foswiki-1.1.2/data/System/InterWikis.txt
< Foswiki-1.1.2/data/System/WebPreferences.txt
< Foswiki-1.1.2/data/System/NewUserTemplate.txt
< Foswiki-1.1.2/data/System/WebTopicEditTemplate.txt
< Foswiki-1.1.2/data/Trash/TrashAttachment.txt
< Foswiki-1.1.2/data/Trash/WebPreferences.txt
< Foswiki-1.1.2/index.html

-- GeorgeClark - 19 Dec 2010

Based on this list, I doubt that a 404 handler to install missing topics would be the right approach. Unless the 404 on Main/WebHome which would be typically the first accessed topic after configure caused all missing files to be populated. The tailoring of the upgrade package is more significant than I realized.

-- GeorgeClark - 20 Dec 2010

There is no exact match of the diff between upgrade package and full package - and the normally tailored topics.

I build the packages using a selection that is also created so that I can more safely build them. They are built manually going through a simple check list.

I do not think a 404 based solution is a good idea either. It has to be a mechanism where you trigger some first time use action. Similar to first time registration which triggers the creation of the WikiUsers topic.

-- KennethLavrsen - 20 Dec 2010