bos@559: <!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : --> bos@559: bos@559: <chapter id="chap:mq-collab"> bos@572: <?dbhtml filename="advanced-uses-of-mercurial-queues.html"?> bos@559: <title>Advanced uses of Mercurial Queues</title> bos@559: bos@584: <para id="x_15d">While it's easy to pick up straightforward uses of Mercurial bos@559: Queues, use of a little discipline and some of MQ's less bos@559: frequently used capabilities makes it possible to work in bos@559: complicated development environments.</para> bos@559: bos@584: <para id="x_15e">In this chapter, I will use as an example a technique I have bos@559: used to manage the development of an Infiniband device driver for bos@559: the Linux kernel. The driver in question is large (at least as bos@559: drivers go), with 25,000 lines of code spread across 35 source bos@559: files. It is maintained by a small team of developers.</para> bos@559: bos@584: <para id="x_15f">While much of the material in this chapter is specific to bos@559: Linux, the same principles apply to any code base for which you're bos@559: not the primary owner, and upon which you need to do a lot of bos@559: development.</para> bos@559: bos@559: <sect1> bos@559: <title>The problem of many targets</title> bos@559: bos@584: <para id="x_160">The Linux kernel changes rapidly, and has never been bos@559: internally stable; developers frequently make drastic changes bos@559: between releases. This means that a version of the driver that bos@559: works well with a particular released version of the kernel will bos@559: not even <emphasis>compile</emphasis> correctly against, bos@559: typically, any other version.</para> bos@559: bos@584: <para id="x_161">To maintain a driver, we have to keep a number of distinct bos@559: versions of Linux in mind.</para> bos@559: <itemizedlist> bos@584: <listitem><para id="x_162">One target is the main Linux kernel development bos@559: tree. Maintenance of the code is in this case partly shared bos@559: by other developers in the kernel community, who make bos@559: <quote>drive-by</quote> modifications to the driver as they bos@559: develop and refine kernel subsystems.</para> bos@559: </listitem> bos@584: <listitem><para id="x_163">We also maintain a number of bos@559: <quote>backports</quote> to older versions of the Linux bos@559: kernel, to support the needs of customers who are running bos@559: older Linux distributions that do not incorporate our bos@559: drivers. (To <emphasis>backport</emphasis> a piece of code bos@559: is to modify it to work in an older version of its target bos@559: environment than the version it was developed for.)</para> bos@559: </listitem> bos@584: <listitem><para id="x_164">Finally, we make software releases on a schedule bos@559: that is necessarily not aligned with those used by Linux bos@559: distributors and kernel developers, so that we can deliver bos@559: new features to customers without forcing them to upgrade bos@559: their entire kernels or distributions.</para> bos@559: </listitem></itemizedlist> bos@559: bos@559: <sect2> bos@559: <title>Tempting approaches that don't work well</title> bos@559: bos@584: <para id="x_165">There are two <quote>standard</quote> ways to maintain a bos@559: piece of software that has to target many different bos@559: environments.</para> bos@559: bos@584: <para id="x_166">The first is to maintain a number of branches, each bos@559: intended for a single target. The trouble with this approach bos@559: is that you must maintain iron discipline in the flow of bos@559: changes between repositories. A new feature or bug fix must bos@559: start life in a <quote>pristine</quote> repository, then bos@559: percolate out to every backport repository. Backport changes bos@559: are more limited in the branches they should propagate to; a bos@559: backport change that is applied to a branch where it doesn't bos@559: belong will probably stop the driver from compiling.</para> bos@559: bos@584: <para id="x_167">The second is to maintain a single source tree filled with bos@559: conditional statements that turn chunks of code on or off bos@559: depending on the intended target. Because these bos@559: <quote>ifdefs</quote> are not allowed in the Linux kernel bos@559: tree, a manual or automatic process must be followed to strip bos@559: them out and yield a clean tree. A code base maintained in bos@559: this fashion rapidly becomes a rat's nest of conditional bos@559: blocks that are difficult to understand and maintain.</para> bos@559: bos@584: <para id="x_168">Neither of these approaches is well suited to a situation bos@559: where you don't <quote>own</quote> the canonical copy of a bos@559: source tree. In the case of a Linux driver that is bos@559: distributed with the standard kernel, Linus's tree contains bos@559: the copy of the code that will be treated by the world as bos@559: canonical. The upstream version of <quote>my</quote> driver bos@559: can be modified by people I don't know, without me even bos@559: finding out about it until after the changes show up in bos@559: Linus's tree.</para> bos@559: bos@584: <para id="x_169">These approaches have the added weakness of making it bos@559: difficult to generate well-formed patches to submit bos@559: upstream.</para> bos@559: bos@584: <para id="x_16a">In principle, Mercurial Queues seems like a good candidate bos@559: to manage a development scenario such as the above. While bos@559: this is indeed the case, MQ contains a few added features that bos@559: make the job more pleasant.</para> bos@559: bos@559: </sect2> bos@559: </sect1> bos@559: <sect1> bos@559: <title>Conditionally applying patches with guards</title> bos@559: bos@584: <para id="x_16b">Perhaps the best way to maintain sanity with so many targets bos@559: is to be able to choose specific patches to apply for a given bos@559: situation. MQ provides a feature called <quote>guards</quote> bos@559: (which originates with quilt's <literal>guards</literal> bos@559: command) that does just this. To start off, let's create a bos@567: simple repository for experimenting in.</para> bos@567: bos@567: &interaction.mq.guards.init; bos@567: bos@584: <para id="x_16c">This gives us a tiny repository that contains two patches bos@567: that don't have any dependencies on each other, because they bos@567: touch different files.</para> bos@559: bos@584: <para id="x_16d">The idea behind conditional application is that you can bos@559: <quote>tag</quote> a patch with a <emphasis>guard</emphasis>, bos@559: which is simply a text string of your choosing, then tell MQ to bos@559: select specific guards to use when applying patches. MQ will bos@559: then either apply, or skip over, a guarded patch, depending on bos@559: the guards that you have selected.</para> bos@559: bos@584: <para id="x_16e">A patch can have an arbitrary number of guards; each one is bos@559: <emphasis>positive</emphasis> (<quote>apply this patch if this bos@559: guard is selected</quote>) or <emphasis>negative</emphasis> bos@559: (<quote>skip this patch if this guard is selected</quote>). A bos@559: patch with no guards is always applied.</para> bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>Controlling the guards on a patch</title> bos@559: bos@584: <para id="x_16f">The <command role="hg-ext-mq">qguard</command> command lets bos@559: you determine which guards should apply to a patch, or display bos@559: the guards that are already in effect. Without any arguments, it bos@567: displays the guards on the current topmost patch.</para> bos@567: bos@567: &interaction.mq.guards.qguard; bos@567: bos@584: <para id="x_170">To set a positive guard on a patch, prefix the name of the bos@567: guard with a <quote><literal>+</literal></quote>.</para> bos@567: bos@567: &interaction.mq.guards.qguard.pos; bos@567: bos@584: <para id="x_171">To set a negative guard bos@559: on a patch, prefix the name of the guard with a bos@567: <quote><literal>-</literal></quote>.</para> bos@567: bos@567: &interaction.mq.guards.qguard.neg; bos@559: bos@706: <para id="x_74a">Notice that we prefixed the arguments to the <command>hg bos@706: qguard</command> command with a <literal>--</literal> here, so bos@706: that Mercurial would not interpret the text bos@706: <literal>-quux</literal> as an option.</para> bos@706: bos@559: <note> bos@706: <title>Setting vs. modifying</title> bos@706: bos@584: <para id="x_172"> The <command role="hg-ext-mq">qguard</command> command bos@559: <emphasis>sets</emphasis> the guards on a patch; it doesn't bos@559: <emphasis>modify</emphasis> them. What this means is that if bos@559: you run <command role="hg-cmd">hg qguard +a +b</command> on a bos@559: patch, then <command role="hg-cmd">hg qguard +c</command> on bos@559: the same patch, the <emphasis>only</emphasis> guard that will bos@559: be set on it afterwards is <literal>+c</literal>.</para> bos@559: </note> bos@559: bos@584: <para id="x_173">Mercurial stores guards in the <filename bos@559: role="special">series</filename> file; the form in which they bos@559: are stored is easy both to understand and to edit by hand. (In bos@559: other words, you don't have to use the <command bos@559: role="hg-ext-mq">qguard</command> command if you don't want bos@559: to; it's okay to simply edit the <filename bos@567: role="special">series</filename> file.)</para> bos@567: bos@567: &interaction.mq.guards.series; bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>Selecting the guards to use</title> bos@559: bos@584: <para id="x_174">The <command role="hg-ext-mq">qselect</command> command bos@559: determines which guards are active at a given time. The effect bos@559: of this is to determine which patches MQ will apply the next bos@559: time you run <command role="hg-ext-mq">qpush</command>. It has bos@559: no other effect; in particular, it doesn't do anything to bos@559: patches that are already applied.</para> bos@559: bos@584: <para id="x_175">With no arguments, the <command bos@559: role="hg-ext-mq">qselect</command> command lists the guards bos@559: currently in effect, one per line of output. Each argument is bos@567: treated as the name of a guard to apply.</para> bos@567: bos@567: &interaction.mq.guards.qselect.foo; bos@567: bos@584: <para id="x_176">In case you're interested, the currently selected guards are bos@567: stored in the <filename role="special">guards</filename> file.</para> bos@567: bos@567: &interaction.mq.guards.qselect.cat; bos@567: bos@584: <para id="x_177">We can see the effect the selected guards have when we run bos@567: <command role="hg-ext-mq">qpush</command>.</para> bos@567: bos@567: &interaction.mq.guards.qselect.qpush; bos@559: bos@584: <para id="x_178">A guard cannot start with a bos@559: <quote><literal>+</literal></quote> or bos@559: <quote><literal>-</literal></quote> character. The name of a bos@559: guard must not contain white space, but most other characters bos@559: are acceptable. If you try to use a guard with an invalid name, bos@567: MQ will complain:</para> bos@567: bos@567: &interaction.mq.guards.qselect.error; bos@567: bos@584: <para id="x_179">Changing the selected guards changes the patches that are bos@567: applied.</para> bos@567: bos@567: &interaction.mq.guards.qselect.quux; bos@567: bos@584: <para id="x_17a">You can see in the example below that negative guards take bos@567: precedence over positive guards.</para> bos@567: bos@567: &interaction.mq.guards.qselect.foobar; bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>MQ's rules for applying patches</title> bos@559: bos@584: <para id="x_17b">The rules that MQ uses when deciding whether to apply a bos@559: patch are as follows.</para> bos@559: <itemizedlist> bos@584: <listitem><para id="x_17c">A patch that has no guards is always bos@559: applied.</para> bos@559: </listitem> bos@584: <listitem><para id="x_17d">If the patch has any negative guard that matches bos@559: any currently selected guard, the patch is skipped.</para> bos@559: </listitem> bos@584: <listitem><para id="x_17e">If the patch has any positive guard that matches bos@559: any currently selected guard, the patch is applied.</para> bos@559: </listitem> bos@584: <listitem><para id="x_17f">If the patch has positive or negative guards, bos@559: but none matches any currently selected guard, the patch is bos@559: skipped.</para> bos@559: </listitem></itemizedlist> bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>Trimming the work environment</title> bos@559: bos@584: <para id="x_180">In working on the device driver I mentioned earlier, I don't bos@559: apply the patches to a normal Linux kernel tree. Instead, I use bos@559: a repository that contains only a snapshot of the source files bos@559: and headers that are relevant to Infiniband development. This bos@559: repository is 1% the size of a kernel repository, so it's easier bos@559: to work with.</para> bos@559: bos@584: <para id="x_181">I then choose a <quote>base</quote> version on top of which bos@559: the patches are applied. This is a snapshot of the Linux kernel bos@559: tree as of a revision of my choosing. When I take the snapshot, bos@559: I record the changeset ID from the kernel repository in the bos@559: commit message. Since the snapshot preserves the bos@559: <quote>shape</quote> and content of the relevant parts of the bos@559: kernel tree, I can apply my patches on top of either my tiny bos@559: repository or a normal kernel tree.</para> bos@559: bos@584: <para id="x_182">Normally, the base tree atop which the patches apply should bos@559: be a snapshot of a very recent upstream tree. This best bos@559: facilitates the development of patches that can easily be bos@559: submitted upstream with few or no modifications.</para> bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>Dividing up the <filename role="special">series</filename> bos@559: file</title> bos@559: bos@584: <para id="x_183">I categorise the patches in the <filename bos@559: role="special">series</filename> file into a number of logical bos@559: groups. Each section of like patches begins with a block of bos@559: comments that describes the purpose of the patches that bos@559: follow.</para> bos@559: bos@584: <para id="x_184">The sequence of patch groups that I maintain follows. The bos@559: ordering of these groups is important; I'll describe why after I bos@559: introduce the groups.</para> bos@559: <itemizedlist> bos@584: <listitem><para id="x_185">The <quote>accepted</quote> group. Patches that bos@559: the development team has submitted to the maintainer of the bos@559: Infiniband subsystem, and which he has accepted, but which bos@559: are not present in the snapshot that the tiny repository is bos@559: based on. These are <quote>read only</quote> patches, bos@559: present only to transform the tree into a similar state as bos@559: it is in the upstream maintainer's repository.</para> bos@559: </listitem> bos@584: <listitem><para id="x_186">The <quote>rework</quote> group. Patches that I bos@559: have submitted, but that the upstream maintainer has bos@559: requested modifications to before he will accept bos@559: them.</para> bos@559: </listitem> bos@584: <listitem><para id="x_187">The <quote>pending</quote> group. Patches that bos@559: I have not yet submitted to the upstream maintainer, but bos@559: which we have finished working on. These will be <quote>read bos@559: only</quote> for a while. If the upstream maintainer bos@559: accepts them upon submission, I'll move them to the end of bos@559: the <quote>accepted</quote> group. If he requests that I bos@559: modify any, I'll move them to the beginning of the bos@559: <quote>rework</quote> group.</para> bos@559: </listitem> bos@584: <listitem><para id="x_188">The <quote>in progress</quote> group. Patches bos@559: that are actively being developed, and should not be bos@559: submitted anywhere yet.</para> bos@559: </listitem> bos@584: <listitem><para id="x_189">The <quote>backport</quote> group. Patches that bos@559: adapt the source tree to older versions of the kernel bos@559: tree.</para> bos@559: </listitem> bos@584: <listitem><para id="x_18a">The <quote>do not ship</quote> group. Patches bos@559: that for some reason should never be submitted upstream. bos@559: For example, one such patch might change embedded driver bos@559: identification strings to make it easier to distinguish, in bos@559: the field, between an out-of-tree version of the driver and bos@559: a version shipped by a distribution vendor.</para> bos@559: </listitem></itemizedlist> bos@559: bos@584: <para id="x_18b">Now to return to the reasons for ordering groups of patches bos@559: in this way. We would like the lowest patches in the stack to bos@559: be as stable as possible, so that we will not need to rework bos@559: higher patches due to changes in context. Putting patches that bos@559: will never be changed first in the <filename bos@559: role="special">series</filename> file serves this bos@559: purpose.</para> bos@559: bos@584: <para id="x_18c">We would also like the patches that we know we'll need to bos@559: modify to be applied on top of a source tree that resembles the bos@559: upstream tree as closely as possible. This is why we keep bos@559: accepted patches around for a while.</para> bos@559: bos@584: <para id="x_18d">The <quote>backport</quote> and <quote>do not ship</quote> bos@559: patches float at the end of the <filename bos@559: role="special">series</filename> file. The backport patches bos@559: must be applied on top of all other patches, and the <quote>do bos@559: not ship</quote> patches might as well stay out of harm's bos@559: way.</para> bos@559: bos@559: </sect1> bos@559: <sect1> bos@559: <title>Maintaining the patch series</title> bos@559: bos@584: <para id="x_18e">In my work, I use a number of guards to control which bos@559: patches are to be applied.</para> bos@559: bos@559: <itemizedlist> bos@584: <listitem><para id="x_18f"><quote>Accepted</quote> patches are guarded with bos@559: <literal>accepted</literal>. I enable this guard most of bos@559: the time. When I'm applying the patches on top of a tree bos@559: where the patches are already present, I can turn this patch bos@559: off, and the patches that follow it will apply bos@559: cleanly.</para> bos@559: </listitem> bos@584: <listitem><para id="x_190">Patches that are <quote>finished</quote>, but bos@559: not yet submitted, have no guards. If I'm applying the bos@559: patch stack to a copy of the upstream tree, I don't need to bos@559: enable any guards in order to get a reasonably safe source bos@559: tree.</para> bos@559: </listitem> bos@584: <listitem><para id="x_191">Those patches that need reworking before being bos@559: resubmitted are guarded with bos@559: <literal>rework</literal>.</para> bos@559: </listitem> bos@584: <listitem><para id="x_192">For those patches that are still under bos@559: development, I use <literal>devel</literal>.</para> bos@559: </listitem> bos@584: <listitem><para id="x_193">A backport patch may have several guards, one bos@559: for each version of the kernel to which it applies. For bos@559: example, a patch that backports a piece of code to 2.6.9 bos@559: will have a <literal>2.6.9</literal> guard.</para> bos@559: </listitem></itemizedlist> bos@584: <para id="x_194">This variety of guards gives me considerable flexibility in bos@559: determining what kind of source tree I want to end up with. For bos@559: most situations, the selection of appropriate guards is bos@559: automated during the build process, but I can manually tune the bos@559: guards to use for less common circumstances.</para> bos@559: bos@559: <sect2> bos@559: <title>The art of writing backport patches</title> bos@559: bos@584: <para id="x_195">Using MQ, writing a backport patch is a simple process. bos@559: All such a patch has to do is modify a piece of code that uses bos@559: a kernel feature not present in the older version of the bos@559: kernel, so that the driver continues to work correctly under bos@559: that older version.</para> bos@559: bos@584: <para id="x_196">A useful goal when writing a good backport patch is to bos@559: make your code look as if it was written for the older version bos@559: of the kernel you're targeting. The less obtrusive the patch, bos@559: the easier it will be to understand and maintain. If you're bos@559: writing a collection of backport patches to avoid the bos@559: <quote>rat's nest</quote> effect of lots of bos@559: <literal>#ifdef</literal>s (hunks of source code that are only bos@559: used conditionally) in your code, don't introduce bos@559: version-dependent <literal>#ifdef</literal>s into the patches. bos@559: Instead, write several patches, each of which makes bos@559: unconditional changes, and control their application using bos@559: guards.</para> bos@559: bos@584: <para id="x_197">There are two reasons to divide backport patches into a bos@559: distinct group, away from the <quote>regular</quote> patches bos@559: whose effects they modify. The first is that intermingling the bos@559: two makes it more difficult to use a tool like the <literal bos@559: role="hg-ext">patchbomb</literal> extension to automate the bos@559: process of submitting the patches to an upstream maintainer. bos@559: The second is that a backport patch could perturb the context bos@559: in which a subsequent regular patch is applied, making it bos@559: impossible to apply the regular patch cleanly bos@559: <emphasis>without</emphasis> the earlier backport patch bos@559: already being applied.</para> bos@559: bos@559: </sect2> bos@559: </sect1> bos@559: <sect1> bos@559: <title>Useful tips for developing with MQ</title> bos@559: bos@559: <sect2> bos@559: <title>Organising patches in directories</title> bos@559: bos@584: <para id="x_198">If you're working on a substantial project with MQ, it's bos@559: not difficult to accumulate a large number of patches. For bos@559: example, I have one patch repository that contains over 250 bos@559: patches.</para> bos@559: bos@584: <para id="x_199">If you can group these patches into separate logical bos@559: categories, you can if you like store them in different bos@559: directories; MQ has no problems with patch names that contain bos@559: path separators.</para> bos@559: bos@559: </sect2> bos@559: <sect2 id="mq-collab:tips:interdiff"> bos@559: <title>Viewing the history of a patch</title> bos@559: bos@584: <para id="x_19a">If you're developing a set of patches over a long time, bos@559: it's a good idea to maintain them in a repository, as bos@592: discussed in <xref linkend="sec:mq:repo"/>. If you do bos@559: so, you'll quickly bos@559: discover that using the <command role="hg-cmd">hg bos@559: diff</command> command to look at the history of changes to bos@559: a patch is unworkable. This is in part because you're looking bos@559: at the second derivative of the real code (a diff of a diff), bos@559: but also because MQ adds noise to the process by modifying bos@559: time stamps and directory names when it updates a bos@559: patch.</para> bos@559: bos@584: <para id="x_19b">However, you can use the <literal bos@559: role="hg-ext">extdiff</literal> extension, which is bundled bos@559: with Mercurial, to turn a diff of two versions of a patch into bos@559: something readable. To do this, you will need a third-party bos@559: package called <literal role="package">patchutils</literal> bos@559: <citation>web:patchutils</citation>. This provides a command bos@559: named <command>interdiff</command>, which shows the bos@559: differences between two diffs as a diff. Used on two versions bos@559: of the same diff, it generates a diff that represents the diff bos@559: from the first to the second version.</para> bos@559: bos@584: <para id="x_19c">You can enable the <literal bos@559: role="hg-ext">extdiff</literal> extension in the usual way, bos@559: by adding a line to the <literal bos@559: role="rc-extensions">extensions</literal> section of your bos@580: <filename role="special">~/.hgrc</filename>.</para> bos@580: <programlisting>[extensions] bos@580: extdiff =</programlisting> bos@584: <para id="x_19d">The <command>interdiff</command> command expects to be bos@559: passed the names of two files, but the <literal bos@559: role="hg-ext">extdiff</literal> extension passes the program bos@559: it runs a pair of directories, each of which can contain an bos@559: arbitrary number of files. We thus need a small program that bos@559: will run <command>interdiff</command> on each pair of files in bos@559: these two directories. This program is available as <filename bos@559: role="special">hg-interdiff</filename> in the <filename bos@559: class="directory">examples</filename> directory of the bos@559: source code repository that accompanies this book. <!-- bos@559: &example.hg-interdiff; --></para> bos@559: bos@584: <para id="x_19e">With the <filename role="special">hg-interdiff</filename> bos@559: program in your shell's search path, you can run it as bos@559: follows, from inside an MQ patch directory:</para> bos@580: <programlisting>hg extdiff -p hg-interdiff -r A:B my-change.patch</programlisting> bos@584: <para id="x_19f">Since you'll probably want to use this long-winded command bos@559: a lot, you can get <literal role="hg-ext">hgext</literal> to bos@559: make it available as a normal Mercurial command, again by bos@580: editing your <filename bos@580: role="special">~/.hgrc</filename>.</para> bos@580: <programlisting>[extdiff] bos@580: cmd.interdiff = hg-interdiff</programlisting> bos@584: <para id="x_1a0">This directs <literal role="hg-ext">hgext</literal> to bos@559: make an <literal>interdiff</literal> command available, so you bos@559: can now shorten the previous invocation of <command bos@559: role="hg-ext-extdiff">extdiff</command> to something a bos@559: little more wieldy.</para> bos@580: <programlisting>hg interdiff -r A:B my-change.patch</programlisting> bos@559: bos@559: <note> bos@584: <para id="x_1a1"> The <command>interdiff</command> command works well bos@559: only if the underlying files against which versions of a bos@559: patch are generated remain the same. If you create a patch, bos@559: modify the underlying files, and then regenerate the patch, bos@559: <command>interdiff</command> may not produce useful bos@559: output.</para> bos@559: </note> bos@559: bos@584: <para id="x_1a2">The <literal role="hg-ext">extdiff</literal> extension is bos@559: useful for more than merely improving the presentation of MQ bos@592: patches. To read more about it, go to <xref bos@559: linkend="sec:hgext:extdiff"/>.</para> bos@559: bos@559: </sect2> bos@559: </sect1> bos@559: </chapter> bos@559: bos@559: <!-- bos@559: local variables: bos@559: sgml-parent-document: ("00book.xml" "book" "chapter") bos@559: end: bos@559: -->