Feature Proposal: migrate from Subversion to git

Motivation

Subversion seems to be annoying a high proportion of our contributors, and many of them are going through the pain of using git svn.

To Do

It was agreed at FoswikiCamp2011 that this is a now, in principle, a do it.

I (JulianLevens) have not marked this as AcceptedProposal, various issues will still need to be resolved and agreed. For example, we discussed using GitHub at the camp and realised that this would not provide support for our need for pre-commit hooks. Therefore, we will will host our own server gain this control. Other options may still needs to be discussed and agreed.

See the SCM Component for the list of related tasks. Please create specific tasks with SCM as a component to enable tracking of the various parts of this project.

Description and Documentation

There are a number of options:

1. host our own Git server
2. GitHub recently added subversion commit support. While they are a closed source commercial service (free for opensource for now) they have alot of mindshare.
3. gitorious - an opensource git hoster
4. sourceforge - they have git, and in many ways it doesn't really matter where the 'master repos' is

Impact

1. we'd need to re-write our tools and docco - unless we use something like github svn support
2. it'll take time and a little risk to retrain some older brains
3. the move from cvs to svn took about 6months before it was decently settled - but at that point, fewer of the devs had used svn - many of our currently active devs already use git for something

It'd be good to have a scriptable interface for commit permissions (SF in its usual style is remarkably poor in this area - web UI only)

Development model

Git supports pretty much any development model. Because it is so trivial to create your own publicly shareable branches, it seems to me that the Foswiki project itself should minimize its workload, and only have the same kind of centralized development in the 'official' repository that we have in svn - trunk, release branches and release tags. This is already enough confusion for users that are not scm literate developers, but tracks the minimally bounded core focus - manage foswiki releases.

One improvement that could become interesting, is to discourage direct pushes to the master repository, and instead, have developers send a pull request to the Smoke Test farm (a set of systems with different Perl's, OS's and configurations), and if a changeset passes, the System pulls that changeset in.

That would ensure that everyone can sync to a codebase that has been tested as effectively as possible...

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

PaulHarvey has been experimenting to make a git (svn) repo per extension. Attached is this quick & dirty script which does the work (doesn't actually download from svn - you have to do git svn fetch --all on each directory). You end up with a directory structure very similar to a trunk checkout, except that each extension is its own separate git-svn repo, with ability to switch between scratch/release/trunk branches, and dcommit/svn rebase from them independently.

And it's fast - all three branches of TinyMCEPlugin come to a 32MB .git directory.

The good:

• git log shows only TinyMCEPlugin history
• Switching branches is instant
• You can easily switch the branch of an individual plugin, rather than the whole checkout
• You can easily switch the branch of core, rather than the whole checkout
• If you branch an extension locally, you only have to deal with changes pertaining to the extension itself, rather than managing a branch of the whole Foswiki trunk

The challenges:

• Commit ids are all changed compared to traditional git svn clone -s checkout.
• All ~500 extensions consume 1011MB in bare form, whereas the .git on an average svn.foswiki.org clone is only around ~250MB
• pseudo-install changes?
• submodule strategy

We need Olivier to put these up as submodules of the foswiki github project; and then do another script which can pull from github and convert the github modules back to git-svn repos (I am mostly interested in having git checkouts which we can dcommit back to svn with).

-- Contributors: SvenDowideit - 14 Jun 2010

Discussion

clearly, I've given up on bzr - it started to annoy me, and as I seem to have begun using github for other things, its an annoyance/context change i don't care for.

oddly, tortoiseGIT doesn't like the github repository ( fatal: http://github.com/foswiki/foswiki/info/refs not found: did you run git update-server-info on the server? ) but is ok with the gitorious one.

-- SvenDowideit - 14 Jun 2010

I like the "social coding" aspect of github. TinyMCE (and many other projects) have picked up quite a few contributions from random forkers there. As you know, spocke even accepted a patch from me, where previously it sat unnoticed on sf.net for over 6 months.

Do we lose much if github/gitorious is merely tracking a hypothetical git.foswiki.org repo? Sounds like we need pros/cons for self-hosted vs gitorious vs github.

My thoughts:

• We should probably continue to host our own repo, unless there's some fantastic benefit that can't be had otherwise.
• We should definitely maintain github/gitorious/etc presence to encourage contributors.
• Would this mean nobody able to commit with svn?
• What's the "easiest" way to migrate svn folk? Has anybody evaluated easygit?

-- PaulHarvey - 14 Jun 2010

I am all for moving to git. svn is regularly taking my machine down for a normal svn up. I really don't care if we push to a github account or not. It will only hold the official trunk to pull from. Everybody can pull from each other as well. So risks are rather low, a lot lower compared to the current single point of failure.

Before moving to git we should think about which development model we should use. While I am pretty sure that best would be to keep a centralized approach (as the github question suggests), we should not discard alternatives too hastily.

-- MichaelDaum - 14 Jun 2010

I had previously opposed a move to git, because windows tools were not available. However, git clients are now available for windows so that argument is moot. I now definitely support a general move to git.

-- MichaelTempest - 14 Jun 2010

I also support such a move, and will be willing to help since I have some experience in moving projects from Subversion to Git.

-- AntonioTerceiro - 14 Jun 2010

I think I don't need to state where I stand on this Just to mention that I tried TortoiseGIT today, and was able to clone the github repo without any issue.

I guess we now need to hear from the "less technical" ones of us (no harm intended, just not everybody is happy with like 10 git commands to do something, but I do love it).

Also, I've written the HowToUseGit, but I agree with MichaelDaum, if we were to move to git, we should define a clear way of working. An easy one is to branch per bug item, commit on the branch, and then merge to trunk once ready.

Finally, all the concerns I've raised about the use of $Rev$ (CDot loves them), and the like, are still valid. Even though I tried to make some hooks where I could to support git, I haven't really tested BuildContrib upload and the like with it, so we might need some dev.

And just to dream... imagine committing your LocalSite.cfg into git, and all your wonderful tools you're using daily but you don't want everybody to look at because you're ashamed of how they're written. Now imagine getting all the latest changes and merging them with your local repo using git pull. It requires some discipline, but I'm pretty confident it's easy to do.

-- OlivierRaginel - 14 Jun 2010

I have never used GIT. I have read about it and I must admit I do not really understand how it works. From the documentation some of you have added it seems to me that there is much more of a learning curve compared to subversion which is extremely simple.

I would not want to fight with a new software CM tool now that we should be focused on getting 1.1 done

And before all you geeks make the decision to do this - please consider how this will impact the occasional extension developer. Will they have an easier learning curve or a steaper learning curve?

You win something with GIT I am sure. But what do we loose? Who do we loose. THINK. Please. And please wait till after 1.1

-- KennethLavrsen - 14 Jun 2010

Hi Kenneth,

I understand you're quite busy at the moment, but probably to partially address your concerns I feel that one of us should evaluate easygit for svn users - it is designed to be as much of a drop-in replacement for svn as possible.

Not sure about windows support.

Git has certainly increased my productivity - I can maintain local ("embarrassing") branches amongst my three staging VMs with ease. In fact, just git clean -fd is a really nice way to clean the checkout compared to some find...exec command.

-- PaulHarvey - 15 Jun 2010

first up, there's no way we do anything before 1.1 is out. As I said, the move from cvs to svn took 6months - there is no way we're changing quickly.

secondly, the occasional extension developer usecase is one of the main things that is slowing down the currently active developers (ok, so Crawford hasn't actually commented ) and we clearly have a strong history of trying to accommodate them (and will continue to do so)

specifically, I don't think what we've done works for the occasional developers either! many have never used svn, and ideally we wouldn't force them to learn any tool.

additional notes:

1. TortoiseGIT has an almost identical use interface to TortoiseCVS and TortoiseSVN so on windows, very little changes
2. I brought up github specifically because they seem to have enabled using svn to commit to the git repositories - to anyone that wants to delay learning git, should (testing needed) be able to work as they used to)

I've added and initial opinion wrt development model up top - please, don't call for discussion, actually discuss

-- SvenDowideit - 15 Jun 2010

@Kenneth, git is not that difficult to handle on normal day-to-day tasks. Even the base commands are named similarly (with some slight semantic changes of course). I am sure you will love it in the end.

Still, this is an essential shift in the tools this project uses. Nobody does that easily. Note also, that today git is rather mainstream as it slowly replaces svn in foss landscape. Not sure if there's any project that would start with svn. They go with one of the distributed VCSes using them in a centralized push development model to lower complexity. Linux-kernel is the exception using a pull model where Linus merges his repo with only a hand full of people he trusts. Distros run their own repos of course pulling from sources they trust from a very wide range of different projects, a modus operandi totally natural for them.

Moving from svn to git is rather a straight forward step that others have accomplished as well. There are little things like $Rev$ to adjust but that's minor. Compare this with the possibilities that unfold for foswiki itself once people get used to git on a broader base.

-- MichaelDaum - 15 Jun 2010

http://github.com/blog/626-announcing-svn-support

and

http://github.com/blog/644-subversion-write-support

neat huh.

-- SvenDowideit - 15 Jun 2010

It seems we all want it but nobody takes the lead. Any CommittedDeveloper?

-- MichaelDaum - 20 Jul 2010

mm, ok, I'll add myself (with date of commitment of my Birthday :0), as I've done several migrations before, including having a hand in the tmwiki cvs->svn one (Where's Ethermage dammit!)

added dream about using a build farm to pull changes into the master repository.

Anyone else that is willing to do the work - please add yourself too :).

That said however, I'd rather see more discussion of impacts, and an idea wrt how&what.. (and how we mitigate the risks)

(And for those not jumping up and down - I don't want to rush this, but subversion is annoying me)

-- SvenDowideit - 20 Jul 2010

This might help with the migration: http://github.com/rcaputo/snerp-vortex

From the author:

Snerp Vortex.

Anagram of Svn Exporter. It's a set of tools I wrote for the POE repository conversion. It converts POE's nearly 2900 revisions in about five minutes, so it's pretty cheap for me to scrap converted repositories and try again. And again.

-- AndrewJones - 29 Aug 2010

Given that we import a lot of javascript code, can I just say that it would be nice to have git to pull from Eg. http://github.com/tinymce/tinymce, http://github.com/jquery/jquery...

I am concerned that we haven't discussed how to utilise git submodules (I've not tried them yet). Possible benefits:

• Extension authors could host their own git repo, but still allow the official foswiki git repo to point to them.
• People checking out a trunk or release branch can choose to only checkout the extensions they are interested in
• With TinyMCE (and other libraries), we get access to all the proper change history of those other projects (git bisect could help us track down bugs introduced in a specific revision of an external project/module), instead of the sledge-hammer that is periodic import of new code.
• ohloh might finally be able to show that we are a perl project, not javascript (once it is clear that the libraries are from 3rd-parties

-- PaulHarvey - 04 Oct 2010

For those who might wonder, I'll try to explain a bit more what submodules are (I use them at work for chef).

Basically, they're pointers to another repository's tree, so, they just contain the SHA1 of the commit-id of the subrepo. Maybe that's not clear enough... Let's say you git submodule add tinymce, this would create a directory called tinymce, and some .gitsubmodule file with the commit id. Then when you want to merge / use a newer version, you go to the tinymce directory, which is a git repository, you git pull to get the latest version of the repo, you checkout the commit you want to use (should be a tag I guess), then you go back to the foswiki dir, and you git commit tinymce, which will simply update the SHA1 link.

So:

• yes, extension authors can host their own git repo like that
• yes, by default one would only get the main repository, and one would need to manually git submodule update $submodule
• Yes, we get access to the history, and we can bisect their repository with our tools, but we would need the sledgehammer to sometimes update the submodule SHA1. Also, bear in mind that some submodules need extra processing before being incorporated into our code. One might need to build or pack them.
• yes, this would make it clear what's our code, and what's YUI, JQuery, TinyMCE and the like

Sven, as this is long overdue, I guess I should add myself as a committed developer and help you out

-- OlivierRaginel - 16 Nov 2010

I am evaluating easygit. I'll see how it goes

-- MichaelTempest - 17 Nov 2010

What I would like to see, beforehand, is a proposal on the development model using pure GIT (not git svn stuff). This document should show both the "casual user" workflow (that will be, at the end, pretty similar to the current workflow) and the "hardcore developer" workflow (which includes branching and stuff... I think they are covered by the HowToUseGit but the workflow is not clear)

This should mitigate any fear of moving to git. The learning curve is steeper than svn, but for a simple workflow you'll end up with using 3 or 4 commands only which are pretty much the same as the svn counterparts.

Anyway, I'm all for moving to git.

-- RafaelAlvarez - 25 Nov 2010

Interposed question from an user: Did anyone try the Trademark Wikis GitPlugin on Foswiki yet? Wouldn't that rise the acceptance of the inevitable considerably?

-- FranzJosefGigler - 26 Nov 2010

Crawford migrated this plugin to Foswiki and uploaded it. He read the code a bit, and stopped. I read it for 5 minutes before concluding whoever wrote that had no idea whatsoever what git was, and its power. So, unless I'm totally wrong (and so is CDot), this plugin should be forgotten and stored in the category: Things you shouldn't do.

To reply to Soronthar, when I wrote HowToUseGit I tried to explain how I was using it, and why. It seems nobody understood, and found it too complex. So I stopped. I agree I should have split it into 2 like you suggest. But any Git SVN crash course would do:

Action	Subversion	git	Remark
Initial checkout	svn checkout url	git clone url	git gets the entire history
Get latest version	svn update	git pull	git gets the new stuff, and tries to merge the tree. SVN merges the files, one by one
Commit	svn commit	git commit -a	git will commit LOCALLY. You have to push your changes back before they're published
Add a new file	svn add	git add	identical
Remove a file	svn rm	git rm	identical
Move a file	svn mv	git mv	identical
Revert a file	svn revert path	git checkout path	identical
Revert a commit	svn merge -c -<rev> svn commit	git revert commit-id;	git revert creates a new commit that removes the changes of the identified commit. svn merge with a negative rev-id removes the rev into the working copy and requires a commit
Get the current status	svn status	git status	identical
Get a diff of changes	svn diff	git diff	identical, but git is faster and more powerful for that, afaik
View history	svn log	git log	identical
Search history of a block of lines	?	git log -L start:end:file	Example: git log -L 2090:2100:lib/Foswiki.spec
Search history of changes to matching regex	?	git log -L :regex:file	Example git log -L :cfg\{SystemWebName:lib/Foswiki.spec
Show an individual commit	svn log -v -r nnnn svn diff -c nnnn	git show commit-id	git can show the complete commit atom with commit messages and the diff of changes using a single command. svn requires 2 commands.
Apply a commit from one branch into another	svn merge -c rev	git cherry-pick commit-id	As for a revert, git commits, whereas SVN does not
Amend file with checkins per line	svn blame	git blame	identical
Find which commit broke something	N/A	git bisect	git can apply checkouts between a "known good" and "known bad" release so the bad commit can be identified.
Publish back your changes	N/A	git push	Everything you do in git is local, so before you push, nobody will see anything. It pushes people to commit often, try things, then revert, and when they're ready to push, they clean their changelog

Feel free to amend / update, I wrote that in 5 minutes

Thanks George for amending. Honestly, reading any crash course should be much better and worth it, but at least you have the basics all in one table. A kind of cheat sheet

-- OlivierRaginel - 26 Nov 2010

I want to address some people who might be put off by reading on IRC about some of the problems myself and perhaps others have had with git... But it's worth noting that in my case, most of the problems I've encountered have been dealing with the git-svn side of things, where if it was a pure git workflow I'd likely not be exposing myself quite so much to git's more boring details.

Having said that, git-svn isn't that difficult to operate, and there's still no way I'd go back to a svn client even if foswiki.org stuck with svn for another 10 years.

-- PaulHarvey - 27 Nov 2010

Thanks to Oliver's excellent comparison git vs svn commands, it should be clear that for a contributor most common steps doesn't change that much at all. However there are still some things to lay down: most importantly building a release.

Another crutial point of moving to git is submodules, as Paul already outlined. What we lose when moving from svn to git is partoal checkouts, that is: using git you will no longer be able to check out a specific subdirectory of your choice anymore. This sounds as a disadvantage at first but stems from a fundamental difference in how these two VCSes work.

That's why there are "submodules" in git.

Before being able to MoveCodeRepositoryToGit, we will ultimately have to define which submodules we want to have and how this is related not only to daily work on the code but also how this is related to the release management and of course branching. I am not sure how switching to a different branch of the main module is propagated to the submodules part of the ckeckout.

It might very well be that even though git has got support for submodules, we will have to ease branching and releasing with a set of management scripts. We already have a bunch of them related to building a release. These need to be rewritten very carefully covering the submodule structure whatever they look like.

Related to slicing the source code into submodules is the directory structure in git: some of the practices we have become familiar working with the svn repo might need a review. For one of course, we will not have to keep branches and tags checked out in separate directories per default. But what does that mean for people that are used to run multiple branches simultaneously as virtual hosts or on different ports or paths from the same checkout area now, e.g. foswiki-1.0.x: port 8000, foswiki-1.1.x: port 8001, foswiki-trunk: port 80, all of these running from their respective checked out branch in a pseudo-install env. How does that look like using git? This also relates to how we will backport fixes in one branch to trunk or vice versa.

The last question is sort of on the fringe of two separate oportunities of using git: (1) code management of an open source project and (2) suporting virtual hosting using git.

As part of (2) it would be even nicer to have foswiki's revision control itself on git as well instead of RCS. So docu and wiki apps you write online are checked in directly and don't need an extra ckeckin to YAFR (yet another f* repository).

-- MichaelDaum - 27 Nov 2010

Michael, some of the stuff you said are either incorrect or misleading, unfortunately.

1. Partial checkout: If you mean that with SVN, you could, from anywhere, simply svn co http://svn.foswiki.org/trunk/SomePlugin, then yes, you cannot do that with git. But as soon as you have a local repository, you can clone it, or checkout whatever you want in whichever branch you want.
2. Several branches. I do that, and I think George explained how I do it in HowToUseGit. Basically, you git clone your main repository (which on linux takes almost no space, as git will create hard links for most of the files in .git/, and then you git checkout the branch you want. For example:
   • git clone github... -- create a directory foswiki with everything in it
   • git clone foswiki master -- to create a local copy. By default it should create hard links if possible. Play with the options if you want more flexibility
   • git clone foswiki release -- to create another local copy
   • cd master && git checkout master -- just to be sure
   • cd ../release && git checkout Release01x01 -- to checkout the release branch
   • git checkout master SomePlugin -- to checkout a plugin residing only in trunk (so non-core), but test it in your release branch

The rest is perfectly sound. The GitPlugin FranzJosef mentionned was targetted exactly at that, but honestly the implementation is so quick & dirty I'd rather rewrite from scratch.

As per the structure, I would push to create a repository for each plugin. Then either we can have core as a special repo, with submodules, or have core as another submodule, and have a special foswiki repo consisting mostly of submodules. It can be seen as way more complex, but git eases the maintenance (something like git submodule update --init --recurse).

-- OlivierRaginel - 27 Nov 2010

Olivier, thanks for the clarification. -- Fixed my name

-- MichaelDaum - 27 Nov 2010

I've just started using repo-per-extension at work. I absolutely think this is the way to go - I suppose the only thing that might be more difficult are the 'global' changes we sometimes do in one commit; these would need a separate commit on every plugin

• Thank God there is git submodule foreach -- OlivierRaginel - 28 Nov 2010

-- PaulHarvey - 28 Nov 2010

Added a script which makes git-repo per extension.

• gitrepoperextension.pl.txt: gitrepoperextension.pl.txt

Let's use this to get each extension up on github as a submodule of foswiki, and then do yet another script so that developers can clone them back with svn tracking properly enabled so we don't lose ability to dcommit back to svn.

-- PaulHarvey - 19 Dec 2010

fun - next up, ./pseudo-install.pl does everything, including get the source, updates things - ala ./pseudo-install.pl rebase developer to update all the developer modules, or ./pseudo-install.pl checkout Release01x01 developer to convert all the developer modules to 1.1, while leaving the other plugins there for testing...

-- SvenDowideit - 19 Dec 2010

http://git.trin.org.au/foswiki/ is now finished sync'ing all ~500 extensions from svn into their own separate git repos, with self-contained scratch/trunk/release branches. There won't be any updates until I have finished the script which will intelligently run git svn fetch --all only on those extensions which have had changes.

-- PaulHarvey - 20 Dec 2010

From discussions with pharvey and babar on IRC:

• git repo organization:
  

• Foswiki core becomes a git super-module.
• pulling down the Foswiki repo gets the core structure, and an empty directory for each Extension
  • git clone --recurse will populate all of the Extensions, similar to a full SVN checkout
• git fetch in the sub-module will populate the sub-module from the repo.
• supermodule points to a specific commit of a sub-module
  • Need to update both the sub-module and super-module to apply a change

Issues?

• Extensions currently use $Rev$ as an increasing rev number. - no equivalent in git.
  • Core extensions change to sync with Foswiki version ex. 1.1.2
  • Core extension released outside of a Foswiki release gets a 4th octet. TinyMCEPlugin 1.1.2.1
  • Non-core extensions need to be converted to a x.x.x release numbering scheme
  • Possible alternative. Use a rest plugin to generate a repo-wide rev number during build.pl release for backwards compatibility
• per-extension fetch is complex for new developers
  • Use the --recurse option to pull the entire repo.
  • Consider adding git support to pseudo-install. So pi of a "empty" directory triggers the fetch to populate the extension
• How do we maintain multiple streams in a sub-module, adding a commit to the Release01x01 version without picking up trunk commits?

-- GeorgeClark - 20 Dec 2010

Branching and sub-modules. After further IRC discussion, each sub-module needs to branch however the pointer from super-module to sub-module is still based on a specific commit. This shows the relationship between the core branches and an individual core extension branches. The only relationship is the commit id. The branch tag is not meaningful when pointing from super to sub module.

All of this is subject to verification. This is somewhat conjecture.

-- GeorgeClark - 21 Dec 2010

Added a quick & dirty script to connect a bunch if git cloned repos back up to svn. Example:

git clone http://git.trin.org.au/foswiki/core.git
git clone http://git.trin.org.au/foswiki/TinyMCEPlugin.git
git clone http://git.trin.org.au/foswiki/TablePlugin.git
./fixclonedrepo.pl *
find . -maxdepth 1 -type d -exec bash -c 'cd {} && git svn rebase' \;

NB: http is a very slow transport for git, need to get these repos up on github.

-- PaulHarvey - 22 Dec 2010

• fixclonedrepo.pl.txt: Fix a git cloned repo so that it can dcommit/svn rebase again

• pseudo-install.pl.txt: a pseudo-install that tries to fetch with git if the module is missing

  ---

Some TODO's ...

• tools/check_manifest.pl is tied to svn commands to list the files in the repository. (Fixed by OlivierRaginel on trunk)
• lib/Foswiki/Net.pm uses the keyword substition of $Rev$ to set the agent string for net requests by Foswiki
• tools/build.pl sets the keyword LASTBUILD - this is not supported by git svn.

One big issue is the svn Keyword expansion on checkout - namely $Rev$ and $Date$.

• $Rev$ is used in the TOPICINFO meta version= string. The foswiki build script replaces it with "1", so it is only of use to developers running on a svn checkout
• $Rev$ and $Date$ are used in the module documentation - as in "API version $Date$ (revision $Rev$)" so that the perldoc tool shows version information.
• $Rev$ is set in many modules $VERSION strings
• $VERSION = '$Date$ $Rev$ ' is set in Foswiki.pm

Crawford requested that we replace the TOPICINFO rev info with a HTML comment in every topic. <!-- Foswiki Checkin: $Date$ --> If we go ahead with this

• Should these comments be removed prior to distribution, like BuildContrib does with the Version= string?

The keywords are substituted by the svn client upon checkout, but git has no comparable capability. How about the following solution:

• For Topics and modules, define a %<macro that can be used by equally by SVN and GIT users to expand into revision information
  • Topics: create a formal document footer: "Foswiki Documentation: Revised: %macro
  • Modules could use it in the pod section for expansion by a plugin during perldoc processing, and in the $VERSION string
• Build.pl would replace these macros with static information from the repo during the build.
• Need to consider Dependency.pm - Dependency checking is the only place I can find $VERSION or $RELEASE being referenced.

Maybe a footer in every Foswiki distributed topic:

Foswiki distribution document: %REVSTAMP% %IF{"'%REVINFO{format="$rev"}%'!='1'" then="Local Revisions"}%

Where REVSTAMP is either dynamic repo information or a static string inserted by build.pl

-- GeorgeClark - 07 Oct 2011

A new tool called http://subgit.com/ might allow us to use both SVN and Git on the same repository, while it keeps them both in-sync. Think of it an alternative to git-svn, but it does all the work on the server, so as far as the user is concerned they are using regular git.

Sounds promising, but haven't tried it myself yet...

-- AndrewJones - 18 May 2012

It appears that subgit is only available for Linux, OSX and Windows. Nothing for FreeBSD where our SVN and Git repo's are located. Other than that, it looked promising.

-- GeorgeClark - 08 Oct 2012

There are only a few places remaining where $Rev is used. They will be removed.

• The Foswiki::Plugins::VERSION will represent all of the PublishedAPI
• The following blog post has good information on VERSION strings: http://www.dagolden.com/index.php/369/version-numbers-should-be-boring/

For Foswiki.pm, I propose we follow the information in the Blog posting, and move to a simple version string as below:

our $VERSION = "1.200";
$VERSION = eval $VERSION;

The end result, there will be two places in Core that has a version string. Foswiki, and Foswiki::Plugins. Individual extensions have already had the $Rev removed

-- GeorgeClark - 09 Oct 2012

Core extensions are another issue. EmptyPlugin documents the following requirements:

# $VERSION is referred to by Foswiki, and is the only global variable that
# *must* exist in this package. This should always be in the format
# $Rev$ so that Foswiki can determine the checked-in status of the
# extension.
our $VERSION = '$Rev$';

# $RELEASE is used in the "Find More Extensions" automation in configure.
# It is a manually maintained string used to identify functionality steps.
# You can use any of the following formats:
# tuple   - a sequence of integers separated by . e.g. 1.2.3. The numbers
#           usually refer to major.minor.patch release or similar. You can
#           use as many numbers as you like e.g. '1' or '1.2.3.4.5'.
# isodate - a date in ISO8601 format e.g. 2009-08-07
# date    - a date in 1 Jun 2009 format. Three letter English month names only.
# Note: it's important that this string is exactly the same in the extension
# topic - if you use %$RELEASE% with BuildContrib this is done automatically.
our $RELEASE = '1.1.2';

Since without SVN, we won't have $Rev$, I suggest we change $VERSION to a real perl version number following the same guidelines. RELEASE can remain the free-form string, as described above.

-- GeorgeClark - 10 Oct 2012

See ConvertToModernPerlVersionStrings for discussion of the VERSION string changes.

---

Now that the modules have been converted to Foswiki version strings, BuildContrib needs to support them instead of generating the SVN Rev. based version string.

For the Foswiki build.pl implementation:

• Prompt - is this a new major or minor build, a rebuild of the previous release, a finalrelease, or the "next in sequence".
  • If a "rebuild" or "test", don't touch the VERSION or RELEASE strings.
  • If major, minor, (or no response), auto-increment the "VERSION" string.
    • v1.1.5 -> v1.1.6_001 -> v1.1.6_002
  • If final, remove the _nnnalpha component of the VERSION.
    • v1.1.6_002 -> v1.1.6
• Prompt - will the release be named? (Alpha, Beta, RC)
  • Derive the "RELEASE" name as Foswiki-<version>-<name>
• Each run that modifies the VERSION and RELEASE strings checks the new version into git or svn.

You are about to build Foswiki.

Enter the type of build.   If this is for your personal use, enter "test"
Enter "major" or "minor" if the major or minor version number should be incremented.
Press enter and the next "alpha" patch version will be assigned.  1.1.6 -> 1.1.7_001.
"release" will remove the alpha designator.   1.1.7_001 -> 1.1.7.

Anything other than "test" or "rebuild" will force a commit of a new version to the respository

$RELEASE is automatically derived from the calculated $VERSION, plus
a name appended for descriptive purposes.

This will be translated to appropriate package and topic names.

I'm now looking in the tags for the designation of the *last* release....

Current version v1.1.6_002, release Foswiki-1.1.6_002-alpha
Is this a new major or minor version, or a final release? (n): 
Do you want to name this release? [y/N] y
Enter optional release name (): RC1 
Building Release: Foswiki-1.1.6_003-RC1 from Version: v1.1.6_003
git commit -m 'Item000: BUILD Foswiki-1.1.6_003-RC1 at Sun Nov 11 03:00:53 2012 GMT' ../lib/Foswiki.pm

---

It would be nice to remind the user to git tag v1.1.6_003 their checkout and then perhaps git push origin v1.1.6_003, or something

-- PaulHarvey - 12 Nov 2012 - 13:55

I just learnt that the git-svn setup as described in HowToUseGit is only an interim solution as long as there is an svn repository hanging around. With it comes the fact that git-svn is a PITA to use and overly complex compared to using git the real way. I can't use git-svn as it is right now. This is too complex for my head. I tried several times. This is not how git is meant to be used. Plain git is what we want. No more svn rebase yadda, please. It really is time to kick svn and move to git for our primary source repository.

-- MichaelDaum - 14 Mar 2013

When we decide against github as the primary repository GITLAB seems to be an excellent tool to cover all of our git needs and more.

-- MichaelDaum - 12 Apr 2013

gitlab looks nice. I still think a github presence is quite imortant though; but should be easy to mirror.

-- PaulHarvey - 12 Apr 2013

Agreed. Note also that a gitlab offers quite a lot of additional features integrated nicely. It could easily replace some of the other tools we are using right now, i.e. trac.f.o and the Tasks web. I didn't have a deep look at it. There is a virtual image at bitnami though ...

-- MichaelDaum - 12 Apr 2013

To do the actual conversion, I just came across: http://john.albin.net/git/convert-subversion-to-git

It seems to be reasonably straightforward, however it creates a single monolithic repo. How can this accommodate the git repo per extension?

-- GeorgeClark - 12 Apr 2013

Example code for a post-commit hook in perl to take care of conversions.

-- MichaelDaum - 28 May 2013

Here's a more thorough solution to tidy the codebase using git hooks https://github.com/mlawren/githook-perltidy

-- MichaelDaum - 09 Jan 2014

It's been 3 years since the 2011 camp agreed in principle to migrate to git. Now that the trunk checkout for foswiki.org is a bit broken, maybe we should move this forward.

Reading a bit more on git hooks, I believe we need both server side hooks, and client side hooks. Client hooks are more helpful to the user, but they are not installed when the project is checked out. They need to be manually installed into the .git/hooks.

See http://git-scm.com/book/en/Customizing-Git-An-Example-Git-Enforced-Policy

Client side hooks:

• Verify that the code is tidy.
• Verify that the commit message is in the correct format
• Query the commit tasks and verify their state can accept checkins

Server side hooks:

• Verify that the user has write access to the tree location? (Core vs. Extension developers)
  • The /home/svn/nextwiki.access file could be converted to Group topics: CoreDevelopersGroup ExtensionDevelopersGroup, ReleaseManagersGroup
• Perform the same checks as the client hooks as a "last resort" to validate checkins in the even the user has not installed the client hooks.
• Email the commit message to the svn list
• Update the commit log in the Task metadata

-- GeorgeClark - 16 Mar 2014

I've created 3 new groups. ReleaseManagersGroup, CoreDevelopersGroup and ExtensionDevelopersGroup. These are mirrored from the nextwiki.access file, but cleaned of duplicates and un-registered names.

The book referenced in the last comment shows an example of controlling push authority using a commit hook. Another thing to look at: https://github.com/sitaramc/gitolite This seems to provide fine grained control over a git repository, but also controls read access. So a hook that queries the Group topics might be more appropriate.

-- GeorgeClark - 16 Mar 2014

To answer my own question on how to query the Item status:

%QUERY{"'Tasks.Item12251'/CurrentState"}%

And fetch it with ?skin=text.

Example: http://foswiki.org/Tasks/ItemStatusQuery?item=Item11267;skin=text

-- GeorgeClark - 16 Mar 2014

Another possible tidy hook: http://www.openswartz.com/2012/09/05/tidying-and-checking-code-on-commit/

I just installed it, but unfortunately it installed 46 cpan dependencies. It does look nice though, doing both Tidy and PerlCritic checks.

-- GeorgeClark - 16 Mar 2014

I've checked in two client side hooks: pre-commit verifies tidy using same code as our svn server side hooks. And commit-msg verifies that properly open Items are identified in the message.

-- GeorgeClark - 17 Mar 2014

There was some more discussion today on IRC as well as some further discovery of our current configuration. MichaelDaum proposed that we consider moving completely to github rather than trying to host our own git server.

I think we need to move this back to "Under Investigation" until some hard decisions are made:

Where do we keep the repo? Do we roll our own git repo, or migrate completely to github, and just keep a backup mirror on foswiki.org I believe the initial plan was to have us maintain our own repo. It gives us better control over commits. We can block push requests for invalid tasks, untidy code, etc. And integrating task updates with checkins is probably simplest this way. But using github as the master is probably more inviting to participation.

There are probably some security implications moving to github. If we want to keep our tasks updated, we'd need to allow github to POST to foswiki.org. They recommend using https://yourUser:yourSecret@yoursite.net/path to secure the post.

Developer	Foswiki Repo	Move to GitHub	Comments
GeorgeClark			Not really sure.	edit
JulianLevens	Y	Y	No, I'm not really sure either	edit

Deployment options Do we structure the repo as:

• core, plus a separate repo per extension,
• Core + default in one repo, Other extensions in a repo per extension (or up to the developer)
• One monolithic repsitory.

There are some discussions above on this. Currently we have a repo per extension on github.

Developer	Repo per Ext	Repo core + non-default	Single monolithic repo	Comments
GeorgeClark		Y		Divide repo's by developer responsibility. Core + default in one, Others per extension or as "owner" desires	edit
JulianLevens			Y	As 1st step. See my comments below	edit
JanKrueger	Y	N	N	Combining multiple things that are worked on more or less indepedently makes merging infeasible or unnecessarily hard. Strongly suggest individual repos and possibly using submodules to pin the right versions together.	edit

Issues Tracker What do we do with the Tasks web. We can continue as-is with the tasks web, enable and use the github trackers, or attempt to do both. If we continue with Tasks, but use github as the repo, we would have to develop a way to accept "POST" updates from github to update task status. See https://help.github.com/articles/creating-webhooks If we do away with Tasks web, how do we migrate & deal with the history.

Developer	Tasks Web	Github Tracker	Comments
GeorgeClark	Y		Keep the Tasks app for now.	edit
JulianLevens	Y		ditto - see my comments below. I even wonder if this has to be either/or, although that creates some sort of synchronisation burden	edit

When do we do this Postpone it again ... and migrate after the 1.2 branch. Or do this as part of the build of the 1.2 branch. or ...

Developer	After 1.2	Branch 1.2 into git	Comments
GeorgeClark		Y	We need to get this done.	edit
JulianLevens		Y	But only if real commitment is obtained to do this within months	edit
JanKrueger		Y		edit

-- GeorgeClark - 21 Mar 2014

I heartily applaud the initiative shown to finally get this done.

Do we have to decide all of this now?

It's not just the resources to achieve all of the above. Even the effort to thrash this out in the community could take time (and at least feel like an endless loop).

How about leaving as a monolithic repo as a 1st step, that doesn't preclude creating separate repos later does it? This would also allow us to consider more carefully what extra repos to create, and indeed create a few repos one at a time at least initially. That way we can even experiment with different approaches and only separate the other extensions with the experience gained.

I partly say this because one repo per extension may may not be ideal. For example:

1. Keep the JQuery stuff in one repo
2. The Wysiwyg and TinyMCE together
3. The MD eco-system

I also note that with git the whole repo can be cloned in minutes with a reasonably fast connection. I believe that svn equivalent is an order of magnitude more.

I further note that there were benefits to me as a new developer to see all those extension directories and appreciate the need not to break these guys. Not to mention that the code in there can provide good learning material. This is an issue of overall visibility and could potentially be solved another way.

My development experience of Foswiki is still rather narrow, but that may make my perspective valuable.

Breaking the git move into pieces may of course cost more resources over the years.

If we can really decide all of the above quickly and commit and deliver within months (3 max?), then then might be an option, but I have doubts.

I would love to see a Camp for all of Foswikis' Gits 2014 and finally do this

-- JulianLevens - 21 Mar 2014

Do we have to decide all this now? Well, it's been 3 years already in the making. I think the easiest time to make the change will be when we branch and start a new release process. There is a lot to change anyway if we stay on svn. We can continue to discuss, but I think at least I'm reaching the point where we need to just try some of this.

Leaving as monolithic PaulHarvey has been working on that a bit. I believe that the github repo is already divided by extension. Actually if I'm understanding what I see on github, it is there as both a monolithic repo, and a per-extension repo.

http://github.com/foswiki/foswiki	is the monolithic repo.
http://github.com/foswiki/_allDeveloper	is "pseudo-monolithic"?
http://github.com/foswiki/core	is the core only
http<://github.com/foswiki/...*...	everything else is each of the extensions as an individual repo

Everything is already maintained on github using a git-svn repo on foswiki.org along with cron jobs that push changes back to github. pharvey has done a large amount of work already on this. For developers, pseudo-install will automatically check out an extension from github if it's not found in the current git repo.

Actually BuildContrib for building foswiki is also mostly converted. It detects a git repo, and generates the appropriate git commit. After build, I just issue a "git svn dcommit" to push to the svn repo. If we use github, that just changes to "git push". I think everything else remains the same. But I won't know for sure until I try it,.

I've started a draft document GitRepository. For now it's mostly a copy of SvnRepository. We'll eventually need it, and some of the workflow and necessary decisions will be easier to figure out as we document the process. As this document is getting long, I think we also need a GitConversionTasks ... which I'll create right now. We can keep the discussion here.

-- GeorgeClark - 21 Mar 2014

Ideas for dealing with Foswikirev links

Once SVN is frozen, we should cache the results of git svn log --oneline --show-commit into a table. Foswikirev could then direct to a page which displays the log record with a link to github. This would avoid updating all the topics to actually point to the github hash entries directly.,

(master)$ git svn log --oneline --show-commit
r17428 | 6a64fa7 | Item12192: using <link /> elems now instead of <style>@import()</style> for better performance
r17427 | ea2e501 | Item12192: better readability on mobile devices
r17426 | ea25103 | Item12192: make it readable on mobile devices
...
r15    | 0b5376a | Adding FastCGIEngineContrib code. Fix to Statistics, so it works without warnings with TSA code. Adding initial experiment about engine-related unit testing
r14    | 7b67798 | Initial import of core trunk code
r1     | 3490bcc | trunk?

-- GeorgeClark - 21 Mar 2014

Something to consider regarding translations. It might be worth reviewing "Weblate" instead of Pootle. It has tighter integration with git, including cross-branch translation. It also has a github service already registered to (from github:)

This service will notify Weblate about pushes to your repository. Weblate will then refresh updated translations and merge them with your changes.

More info: http://weblate.org/

-- GeorgeClark - 21 Mar 2014

I've got a simple extension that maps a svn rev to a github hashref. Invalid svn rev. It loads the map from rev to ref from the DATA segment of a module.

Unfortunately interwiki links don't expand macros because of rendering order. A simple change to call expandCommonMacros() for the link text resolves the issue. See Item12819.

Unfortunately we ship Foswikirev links in our release notes, plugin topics, and other foswiki documents, so this isn't an issue restricted just to foswiki.org. So what started out as a simple plugin for foswiki.org to deal with the Tasks web probably needs to be released.

-- GeorgeClark - 23 Mar 2014

CrawfordCurrie has raised some concerns. IRC Discussion here:

gac410

Hey CDot I've got a plugin that maps svn rev to git hashref. Implements %REV2REF macro. .. But InterWikis doesn't expand macros, so no easy way to convert Foswikirev: links. Needs a plugin change.

CDot

excellent

gac410

I realized this is a bigger issue ... we ship Foswikirev links in many of our documents. release notes, plugin topics,
Item12819

FoswikiBot

http://foswiki.org/Tasks/Item12819 [ Item12819: Pass Interwiki links through expandCommonVariables ]

gac410

So we need to consider shipping the plugin ... or decide how to deal with existing sites / documents. I suppose we can keep trac.foswiki.org around, along with an archived svn repo too.

CDot

CDot isn't getting involved with the git migration. It's not something I particularly want, and I'm concerned that it will damage the project, but I'm a lone voice in the wilderness.

gac410

hmmm. concerns about damage... that certainly needs to be addressed. lone voice doesn't mean it's not a valid voice.

CDot

I've already voiced my opinion, but there are enough voices in favour that I'm staying out of it.

gac410

okay fair enough. Are your concerns in the wiki?

-

gac410 goes to read

CDot

pretty sure they are, but no idea where.

gac410

Okay... I'll look.
hmpf. You are not on the Foswiki:Development/MoveCodeRepositoryToGit which was the proposal. It mentions decision at FoswikiCamp2011, but nothing referenced there or in the release notes.

FoswikiBot

http://foswiki.org/Development/MoveCodeRepositoryToGit [ MoveCodeRepositoryToGit ]

CDot

my basic concern is that git works well when you have a large project with an active core. The active core can act as a "gate" on developments, picking and choosing what to merge with the integration stream.
foswiki is a much smaller affair than that. If we start gitting, I can foresee a rapid explosion of forks as individuals feel their priorities differ from the core stream.
while that can fuel a lot of development, it also poses an imopssible problem for anyone who is trying to create a "golden" distribution - of any part - from the myriad of source trees

gac410

I think we will still have one "official" source tree. The "golden distribution" would only be generated from one single repo consisting of the core + default extensions. People who fork can submit "pull requests" to ask us to incorporate changes.
I could see extensions certainly splitting off. I think that
that's a good thing. Our extensions get rather stale, and tbh I'd rather see us not own the stuff that is of lower quality / orphaned.

CDot

the problem would be sealed the first time someone says "That's fixed in MY distribution". That's what would kill us.

gac410

Unless they also build the Foswiki tgz. If someone else fixes it, we have the opportunity of pulling the fix in, And that is a bit easier process as well. So easier to fork, but easier to merge too.
We've been on github for a couple of years, and have not seen all that much forking.

CDot

no, because out infrastructure doesn't favour it (I think).
but you are changing that. making it easier.

gac410

hm... but a fork goes "off infrastructure" regardless of what we do. We'll still have one golden repo, and release from that single repo. I don;t think that changes at all.

CDot

Maybe I'm needlessly negative. If people can make git work, then I'm all for it. But I have my doubts, and I just don't have (or rather, don't want to have) headspace for it.

gac410

Okay fair enough.

gac410

I'll get this conversation into the migration topic. If nothing else you can raise an "I told you so" later But seriously, we should consider project guidelines and policies to help control / mitigate your concerns.

CDot

indeed

-- GeorgeClark - 23 Mar 2014

CDot's concerns are valid. There are a lot of examples for projects being forked with each repo holding new features never merged together into one. This occurs especially when the original repo hasn't been maintained for a while.

Git makes it especially easy to fork ... and forget.

However git does not necessarily lead towards such a sprawl, which indeed would delude the mainline project. Instead it would be more of a sign of the mainline project not actively merging in other contributions.

Git does not necessarily lead towards forking into separate repos either. While it does support forking, it is not required to make use of this feature per se just to create a "personal repo" more or less driven by single individuals. I'd consider this a bad practice showing a lack of communication between contributors ... basically a misconception of collaboration in an open source environment.

These are all non-technical issues when managing software, but not innate to git. Git allows you to implement whatever regime you'd like to establish:

1. a centralized repo everybody pushes to and pulls from or
2. multiple repos with pull requests from one to the other
3. maybe a mix of both for a certain period of time

Subversion basically only allows (1). Everything else is a pain with it. Large projects might work quite fine using model (2). Small projects suffer more than have any benefit from (2). In terms of lines of code Foswiki is a large project, not so in terms of manpower. Therefore we should go with a model (1) development ... as we allready do with good old subversion. Small projects - in terms of lines of code - might be okay with subversion as a technology. Not so Foswiki. We really are having a problem here. Development suffers from subversion as it hinders productivity for each and every developer.

-- MichaelDaum - 23 Mar 2014

More discussion on IRC tonight about mapping SVN rev's to github hashes. We've discovered

• that there is not a 1:1 relationship from svn rev to git commit. Cross-branch commits on svn result in multiple git commits.
  • Need to use the log from the github clone to get the commits, our git-svn clone doesn't have the same hash, esp. with cross-branch checkins
  • For mapping, when more that one git commit matches the svn rev, keep master as the preference.
• Organization of the repos under the github foswiki account.
  • "foswiki" repo would be the core + default extensions. ( foswiki/foswiki )
  • Other extensions would be each in their own repo. ( foswiki/SomePlugin )
  • Question on whether or not to use submodules. They make more sense when including a specific component / tag into an extension, like TinyMCE into TinyMCEPlugin
  • Mapping then needs to be svn rev => repo/commit/hash
• Tasks web would track stuff in the "foswiki" github account. We won't try to track changes made in other users own repos. They can use the gh tracker, or other tools.
• Extension web changes
  • Developed in SVN: Becomes "Developed In Foswiki" - meaning the foswiki repo on github.
  • We need a field for the location of the source, to point to alternative repos, etc. Also might reflect an extension that is in a repo under foswiki but a different name.

See logs starting http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2014-03-23,Sun&sel=215#l211 and http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2014-03-24,Mon&sel=1#l-3

-- GeorgeClark - 24 Mar 2014

Migration Issues

Objective: Split out the core of the foswiki project, including branches, complete history of moved files. Ideally we would get tags as well, but adding back in lightweight git tags is fine.

Below is what I've tried.

The issue, (described in an email: http://sourceforge.net/p/foswiki/mailman/message/32718315/ is that we will lose the history on any file moved from a core location into an extension. It has to do with the way git tracks history across moves. By moving the file into a different repo, the link to the history in the old location is lost. The only way that I can see to have an intact history of the foswiki codebase is to use a monolithic repository at least for the core + default, including the original locations of files. I've tried several strategies.:

I followed both of the process here: http://stackoverflow.com/questions/19954485/extract-multiple-directories-using-git-filter-branch (note that this runs 4-5 hours).

cat preserve-only.sh 

IFS=':'
GLOBIGNORE="$*"
ls -lad *

cat filertbr.sh 

git filter-branch -f --prune-empty --index-filter 'git rm --ignore-unmatch --cached -r -q -- . ; git reset -q $GIT_COMMIT -- AutoViewTemplatePlugin BehaviourContrib CommentPlugin CompareRevisionsAddOn ConfigureContrib core CpanContrib EditRowPlugin EditTablePlugin EmptyPlugin FamFamFamContrib HistoryPlugin HomePagePlugin InterwikiPlugin JEditableContrib JQueryPlugin JSCalendarContrib MailerContrib NatEditPlugin PatchFoswikiContrib PatternSkin PlainFileStoreContrib PreferencesPlugin RCSStoreContrib RenderListPlugin SlideShowPlugin SmiliesPlugin SpreadSheetPlugin SubscribePlugin TablePlugin TinyMCEPlugin TipsContrib TopicUserMappingContrib TWikiCompatibilityPlugin TwistyPlugin UpdatesPlugin WysiwygPlugin ' -- --all

cat filterbr2.sh
git filter-branch -f --prune-empty --tree-filter 'bash preserve-only.sh AutoViewTemplatePlugin BehaviourContrib CommentPlugin CompareRevisionsAddOn ConfigureContrib core CpanContrib EditRowPlugin EditTablePlugin EmptyPlugin FamFamFamContrib HistoryPlugin HomePagePlugin InterwikiPlugin JEditableContrib JQueryPlugin JSCalendarContrib MailerContrib NatEditPlugin PatchFoswikiContrib PatternSkin PlainFileStoreContrib PreferencesPlugin RCSStoreContrib RenderListPlugin SlideShowPlugin SmiliesPlugin SpreadSheetPlugin SubscribePlugin TablePlugin TinyMCEPlugin TipsContrib TopicUserMappingContrib TWikiCompatibilityPlugin TwistyPlugin UpdatesPlugin WysiwygPlugin' -- --all

The problem here was that even though the trailing -- --all should have included all branches, the result was only the master branch. It also appeared to lose all the svn info. That isn't an issue in itself provided that this process runs after svn is frozen. But if we need to svn rebase, during the conversion window, we need the svn connection. The process

Next attempt was to use svnadmin dump / svndumpfilter / svnadmin load to load a subset (http://stackoverflow.com/questions/11367755/split-a-subversion-repository-project-to-two-git-repositories)

This looked like it might work - the filter would fail if a source of a moved file was omitted, so iteratively the dump got a bit larger, but it looked managable. But there are two issues:

• The load renumbered the revisions even though the filter option to not renumber was requested.
• And the load crashed

svnadmin load foswiki-core   < /var/SVN/filtered.out  > load.log
svn: E235000: In file 'subversion/libsvn_subr/mergeinfo.c' line 1181: assertion failed (IS_VALID_FORWARD_RANGE(second))

At this point I really don't know how to proceed.

See also http://programmers.stackexchange.com/questions/161293/choosing-between-single-or-multiple-projects-in-a-git-repository?rq=1 for some information on submodules vs. monolithic. I think the killer issue for me, beyond loss of history, is the lack of git bisect for the entire core+default structure.

Note from the svn filter, the following directories appear to be required due to moves to/from core:

AutoViewTemplatePlugin BehaviourContrib BuildContrib CommentPlugin CompareRevisionsAddOn ConfigureContrib core CpanContrib EditRowPlugin EditTablePlugin EmptyJQueryPlugin EmptyPlugin FamFamFamContrib HistoryPlugin HomePagePlugin ImageGalleryPlugin InterwikiPlugin JEditableContrib JQueryPlugin JSCalendarContrib MailerContrib NatEditPlugin PatchFoswikiContrib PatternSkin PlainFileStoreContrib PreferencesPlugin QueryAcceleratorPlugin RCSStoreContrib RenderFormPlugin RenderListPlugin SlideShowPlugin SmiliesPlugin SpreadSheetPlugin SubscribePlugin TablePlugin TestFixturePlugin TinyMCEPlugin TipsContrib TopicUserMappingContrib TWikiCompatibilityPlugin TwistyPlugin UnitTestContrib UpdatesPlugin WysiwygPlugin ZonePlugin

-- GeorgeClark - 15 Aug 2014

I've re-run the filter-branch command with a tree-filter and tag-filter cat. Ran nearly 40 hours, but I appear to now have a "foswikiDistribution" repo that I could push to github. History across moved files is complete. That was the breaking issue with the submodule approach. svn info is also intact so I can keep this in sync with svn for a few more updates before we make the cutover.

So if there is agreement, we abandon submodules. We'll have a repo per extension except for the default extensions.

Now I need to redo the sha1 audit between svn and git, and re-extract the commits for all svn revisions. The sha1 commit hashes changed with the rewritten core.

-- GeorgeClark - 17 Aug 2014

It's pushed to github, Still not fully validated: https://github.com/foswiki/distro

History appears complete for work within the foswiki project. For ex, the moves from core to ConfigureContrib, RCSStoreContrib, etc. all are followable. Most of the initial refactoring of TWiki is also tracked.

-- GeorgeClark - 18 Aug 2014

Commenting is disabled while not logged in