Content AND Presentation

2019-08-19 Transplanting a directory to another Git repository

Recently, I had a very specific need. I wanted to move a directory to another Git repo, but I really needed to preserve its history.

There is a quite well-known instance of a similar thing – the famous coolest merge ever is basically importing one project into another, preserving its history. My use-case, however, was a bit more difficult because I wanted to simultaneously move things to a directory with another name. (I could, of course, start with making a temporary clone – or a branch – in the “source” repo, delete everything I do not want to merge in it, change the directory structure to reflect what I want in the “destination” repo and commit all these changes. I wanted to avoid that, though. One of the reasons is that file renaming, while supported by Git, introduces unnecessary complications when analysing history.)

So, let’s get started. Assume that we have two Git repos, source and dest. In the source repo we have (among other things) a subdirectory called directory, and we want to move all files from it to a directory called folder in dest.

To begin with, let us create the repos. (On my machine, /mem is a scratch directory, much like /tmp, with the advantage that it only contains stuff I put here instead of a whole lot of things some random programs decide to save in /tmp. Also, as the name suggests, it resides in a ramdisk, so nothing there sticks for too long.)

cd /mem
rm -rf source dest
git init source
git init dest
cd source
mkdir directory
echo "some file in the root dir" > some-file.txt
echo "another file in the directory dir" > directory/another-file.txt
git add .
git status
git commit -m "Initial commit"
echo "added line" >> directory/another-file.txt
git commit -am "Add a line"
echo "An unrelated commit" >> some-file.txt
git commit -am "An unrelated commit"
echo "A commit spanning everything" >> some-file.txt
echo "A commit spanning everything" >> directory/another-file.txt
git commit -am "Make huge changes"
cd ../dest
echo "The destination repo" > README.txt
git add README.txt
git commit -m "Add README.txt"

We now have two simple Git repos to experiment. (Note that because of rm -rf, the snippet above will recreate them from scratch every time it is run, which is quite convenient for experimentation.)

If we were happy to just merge everything from source to dest, things would be very easy:

git remote add source ../source
git fetch source
git merge source/master --no-edit --allow-unrelated-histories

Note the options for git merge. The man page says explicitly that usually you do not want --no-edit, but since I want a smooth presentation of the main ideas here and not manually crafted merge commit messages, this is exactly what I need. The option --allow-unrelated-histories (which was the default in older versions of Git) is pretty self-explanatory.

This approach works well if we want just to merge two repositories (solving conflicts should they arise, but this is another story), but it is not what we want here. The first problem is that it imports also the some-file.txt, and we only want directory and its contents. (For bonus points, notice how the second commit in source touches both a file in directory and a file outside of it – we would like to perform some surgery on this commit to preserve only the modification to another-file.txt.)

Well, this is Git, so all this is perfectly doable. There is even a dedicated Git command solving a very similar issue, called git-subtree. We will not resort to it, however (I will probably write another post on git-subtree one day), using lower-level git-filter-branch, git-read-tree and a few other commands instead.

In fact, a ready solution for the hard part can be easily found on the Internet. What I aim to do here is to (try to) explain the meaning of the commands involved. Note that, like in my previous post, I found this out by careful experiments and studying the manual, not by reading the Git sources, so there may be mistakes. Please point them out in the comments should you spot any.

So, let’s get down to business. First, we will excise everything but directory from the source repo (this is actually the easy part):

cd /mem
rm -rf source-tmp
git clone source source-tmp
cd source-tmp
git filter-branch --prune-empty --subdirectory-filter directory -- --all

Note how the contents of directory have just migrated to the root directory of our repo. Also, inspect the history and note how the “Make huge changes” commit now only touches the another-file.txt (which is logical, since it has nothing else to touch now, but still nice).

Interestingly, we have given the --all parameter to make Git operate on all the references (not “all the commits”!), local and remote. Without it, the history on branches other than “master” would be unaffected, thus leaving a terrible mess. Also, a good thing to know is that git-filter-branch will create a directory called .git/refs/original, where it stores all references it has changed. This means that the whole operation is easily undoable – just move .git/refs/original/refs to .git/refs, overwriting everything in the process, and you are done. (In particular, we did not really have to create source-tmp – but throwing it away is easier than manipulating stuff within .git.) You may read more about --all in e.g. the manpage of the plumbing command git-rev-list.

Another thing worth mentioning is the --prune-empty switch. Here, things get a bit hazy for me. The manual says that its aim is to remove empty commits (apart from merge commits), but a quick experiment showed that the commands seems to work the same way without it. (I asked about it on StackOverflow and learned that indeed, --prune-empty is superfluous in this case.)

Now we need to import (merge) our temporary repo into dest.

cd /mem/dest
git remote add -f source ../source-tmp
git merge -s ours --no-commit --allow-unrelated-histories source/master

Now this is where the fun starts. First we add our temporary repo as a remote (and immediately fetch from it, using the -f option) – this is simple. Then, we prepare the merge. First of all, we supply the ours merge strategy. (Note: this must not be confused with the ours option for the recursive merge strategy. See the manpage of git-merge for more information.) This means that the “merge” will actually completely disregard the tree in the merged-in heads. In other words, after an ours merge, while the history will look as if two (or more) branches has been merged, all the changes from the merged-in branches will be completely lost. (This may be actually useful in rare cases, I guess.)

The next thing is the --no-commit option. It seems obvious, but actually it is not so in this case. I mean, with a “normal” merge, this just leaves the last step (the actual commit) to the user (much like in the case of conflicts). However, you might wonder what this does in the case of the our stategy we have used. Turns out, the only thing our merge command does is update a few files in the .git directory: ORIG_HEAD (the reference to the head before the merge started – this reference is actually written by more operations in Git so that undoing is easier), MERGE_MSG (pretty obvious), MERGE_MODE (no-ff in our case, which is not surprising) – this seems a bit, erm, underdocumented, but I found some information here, and – most importantly – MERGE_HEAD, which contains the reference to the branch we are merging in (source/master). (In the case of an octopus merge, this file contains more references, of course.)

The --allow-unrelated-histories option we already mentioned, and there is not much to explain here.

So, if we commit now, the commit would be “empty” (i.e., it would not introduce any changes), but the history would show that we have merged the source/master branch (and we would have two roots now). What we need to do is to put the contents of source/master (i.e., the current state of the directory in the source repo) into the folder directory. This is the easier part and can be done with yet another plumbing Git command, read-tree.

cd /mem/dest
mkdir folder
git read-tree --prefix=folder/ -u source/master

Now we could just copy the files from source-tmp and stage them instead. (This is a bit hazy again. I performed an experiment to check if copying and staging from source-tmp would lead to the same result. It did, in the sense that in both cases the .git directory contained almost exactly the same stuff (in particular, the objects were bit by bit the same). The “almost” part was the index file (i.e., the “staging area”). While git ls-files --stage also showed the same output, there were binary differences in .git/index. If some brave soul wants to perform a similar experiment and dig even deeper into this, here is the official description of the index file format. Also, to make sure the objects in both cases are the same so that the comparison is fair, you have to make sure that the timestamps of all commits are the same in both cases. One way to ensure that is to set the environment variables GIT_AUTHOR_DATE and GIT_COMMITTER_DATE (see e.g. here for some explanation) or use the datefudge utility with the -s option, which I did.)

However, git-read-tree does this in one step instead of two: it puts everything from the source/master commit into the index (aka staging area), which is its basic aim, but the -u option makes it also update the working directory. (Again, the man page is not very precise here – it says what the -u switch does after a successful merge. In our case, we do not request a merge, but OTOH we will not have any conflicts, since we assume the folder is empty. I made a few experiments, and it seems that -u is only meaningful with -m, --prefix or --reset. That kind of makes sense, although is not said explicitly in the manual.)

Since we now have everything we want in the index, the only thing that’s left is to commit the changes and delete the temporary repo:

git commit -m "Merge source into dest under folder"
git remote rm source
rm -rf ../source-tmp

And we are done!

Of course, again, this is Git, so this is definitely not the whole story. We could – instead of deleting source-tmp – pull further changes (under the assumption that source is being worked upon, we could repeat the filter-branch stuff in the future and pull the resulting changes into desc. In case you are afraid that this will mess up the history: no, it won’t, git-filter-branch generated perfectly deterministic commit hashes every time you repeat it (which is not surprising, taking into account what exactly goes into a commit hash). Also, Git has the very useful in this case subtree merge strategy (which I admit I haven’t experimented with) which apparently does not even require you to specify the folder explicitly. Also, there is the git-subtree command I mentioned. In any case, the above was enough for me, so I decided to share it.

CategoryEnglish, CategoryBlog, CategoryGit

Comments on this page

2019-08-11 A console.dir hint

As everyone knows, console.log is the ultimate debugging tool. ;-) Joking aside, it is genuinely useful (together with console.error) in scripts. I often write small (and sometimes not so small) CLI utilities in Node.js, and they are really indispensable.

Sometimes, however, you want to output a complicated structure. You can use JSON.stringify, but it’s not always what you want, since some structures may be circular, and JSON.stringify will barf at them. Then, of course, console.dir is your friend.

What if console.dir is not enough? Maybe you need to go deeper than the default two levels. Maybe you want the result to go to stderr instead of stdout. Maybe you want to have the keys sorted. If you find any of those (potentially) useful, check out util.inspect. It turns out that console.dir(...) is equivalent to console.log(util.inspect(...)), so you can console.error(util.inspect(...)) to print to stderr. Also, util.inspect accepts an optional argument, allowing for all things I hinted at above and more. Good to know!

CategoryEnglish, CategoryBlog, CategoryJavaScript

Comments on this page

2019-08-05 datefudge and agenda testing

Some time ago, a question was asked on the Org-mode mailing list about a specific kind of task in Org agenda. This made me think about debugging one’s agenda settings. I’ve already written about batch agenda, but one problem with agenda testing is that it is inherently stateful, in one of the worst ways – it depends on the notion of now.

Debugging time-related stuff is hard. (Well, time-related stuff is hard, after all.) It would be great if you could just manipulate Emacs into thinking that the time is some day in the future (or in the past)…

Well, actually, it can be done – and it’s easier than I thought.

One way would be to advise the current-time function. This is potentially dangerous, but should work for Org-related stuff (since Org-mode is written entirely in Emacs Lisp, there should be no danger of current-time being called by a primitive function written in C – but you can never be sure if Org doesn’t use some primitive calling current-time…)

Perhaps a better way would be just to fset the current-time function to something different. This is not very nice, since we usually only want to mess up with it temporarily – but we can use the cl-flet macro which does just that (or just fire a dedicated Emacs instance).

However, I found a completely different way of doing that, which should be much better than all of the above ideas. Why not change the system time? Of course, I do not want to mess up with system time on my machine – but there are two things I can do.

One is to run Emacs in a virtual machine, where I can do all sorts of system-related stuff and not worry about breaking my computer. For VirtualBox, changing the system time of the guest is easy (see here and here).

But there is even a better thing you can do, at least if you run a GNU/Linux system (as I do). There is this little utility called datefudge, which accepts any shell command and calls it, first modifying a few system calls responsible for getting the current time.

Even better, it has two modes of operation: “normal”, where the system clock appears to run as usual (but starting at the given time), and “static”, where the system clock appears to sit still on the time given.

Now I can easily (and reliably) test my agenda settings with a sample “fake” Org file, saying just something along the lines of datefudge 2019-08-05 emacs -Q --batch .... This way, we can even easily have automated tests for Org agenda.

CategoryEnglish, CategoryBlog, CategoryEmacs, CategoryOrgMode

Comments on this page

More...

CategoryEnglish, CategoryBlog