hgbook
view fr/ch13-mq-collab.xml @ 996:6f8c48362758
merge with trunk
| author | Romain PELISSE <belaran@gmail.com> |
|---|---|
| date | Sat Sep 12 17:58:56 2009 +0200 (2009-09-12) |
| parents | 6b680d569bb4 f0110009e946 |
| children |
line source
1 <!-- vim: set filetype=docbkxml shiftwidth=2 autoindent expandtab tw=77 : -->
3 <chapter id="chap:mq-collab">
4 <?dbhtml filename="advanced-uses-of-mercurial-queues.html"?>
5 <title>Advanced uses of Mercurial Queues</title>
7 <para id="x_15d">While it's easy to pick up straightforward uses of Mercurial
8 Queues, use of a little discipline and some of MQ's less
9 frequently used capabilities makes it possible to work in
10 complicated development environments.</para>
12 <para id="x_15e">In this chapter, I will use as an example a technique I have
13 used to manage the development of an Infiniband device driver for
14 the Linux kernel. The driver in question is large (at least as
15 drivers go), with 25,000 lines of code spread across 35 source
16 files. It is maintained by a small team of developers.</para>
18 <para id="x_15f">While much of the material in this chapter is specific to
19 Linux, the same principles apply to any code base for which you're
20 not the primary owner, and upon which you need to do a lot of
21 development.</para>
23 <sect1>
24 <title>The problem of many targets</title>
26 <para id="x_160">The Linux kernel changes rapidly, and has never been
27 internally stable; developers frequently make drastic changes
28 between releases. This means that a version of the driver that
29 works well with a particular released version of the kernel will
30 not even <emphasis>compile</emphasis> correctly against,
31 typically, any other version.</para>
33 <para id="x_161">To maintain a driver, we have to keep a number of distinct
34 versions of Linux in mind.</para>
35 <itemizedlist>
36 <listitem><para id="x_162">One target is the main Linux kernel development
37 tree. Maintenance of the code is in this case partly shared
38 by other developers in the kernel community, who make
39 <quote>drive-by</quote> modifications to the driver as they
40 develop and refine kernel subsystems.</para>
41 </listitem>
42 <listitem><para id="x_163">We also maintain a number of
43 <quote>backports</quote> to older versions of the Linux
44 kernel, to support the needs of customers who are running
45 older Linux distributions that do not incorporate our
46 drivers. (To <emphasis>backport</emphasis> a piece of code
47 is to modify it to work in an older version of its target
48 environment than the version it was developed for.)</para>
49 </listitem>
50 <listitem><para id="x_164">Finally, we make software releases on a schedule
51 that is necessarily not aligned with those used by Linux
52 distributors and kernel developers, so that we can deliver
53 new features to customers without forcing them to upgrade
54 their entire kernels or distributions.</para>
55 </listitem></itemizedlist>
57 <sect2>
58 <title>Tempting approaches that don't work well</title>
60 <para id="x_165">There are two <quote>standard</quote> ways to maintain a
61 piece of software that has to target many different
62 environments.</para>
64 <para id="x_166">The first is to maintain a number of branches, each
65 intended for a single target. The trouble with this approach
66 is that you must maintain iron discipline in the flow of
67 changes between repositories. A new feature or bug fix must
68 start life in a <quote>pristine</quote> repository, then
69 percolate out to every backport repository. Backport changes
70 are more limited in the branches they should propagate to; a
71 backport change that is applied to a branch where it doesn't
72 belong will probably stop the driver from compiling.</para>
74 <para id="x_167">The second is to maintain a single source tree filled with
75 conditional statements that turn chunks of code on or off
76 depending on the intended target. Because these
77 <quote>ifdefs</quote> are not allowed in the Linux kernel
78 tree, a manual or automatic process must be followed to strip
79 them out and yield a clean tree. A code base maintained in
80 this fashion rapidly becomes a rat's nest of conditional
81 blocks that are difficult to understand and maintain.</para>
83 <para id="x_168">Neither of these approaches is well suited to a situation
84 where you don't <quote>own</quote> the canonical copy of a
85 source tree. In the case of a Linux driver that is
86 distributed with the standard kernel, Linus's tree contains
87 the copy of the code that will be treated by the world as
88 canonical. The upstream version of <quote>my</quote> driver
89 can be modified by people I don't know, without me even
90 finding out about it until after the changes show up in
91 Linus's tree.</para>
93 <para id="x_169">These approaches have the added weakness of making it
94 difficult to generate well-formed patches to submit
95 upstream.</para>
97 <para id="x_16a">In principle, Mercurial Queues seems like a good candidate
98 to manage a development scenario such as the above. While
99 this is indeed the case, MQ contains a few added features that
100 make the job more pleasant.</para>
102 </sect2>
103 </sect1>
104 <sect1>
105 <title>Conditionally applying patches with guards</title>
107 <para id="x_16b">Perhaps the best way to maintain sanity with so many targets
108 is to be able to choose specific patches to apply for a given
109 situation. MQ provides a feature called <quote>guards</quote>
110 (which originates with quilt's <literal>guards</literal>
111 command) that does just this. To start off, let's create a
112 simple repository for experimenting in.</para>
114 &interaction.mq.guards.init;
116 <para id="x_16c">This gives us a tiny repository that contains two patches
117 that don't have any dependencies on each other, because they
118 touch different files.</para>
120 <para id="x_16d">The idea behind conditional application is that you can
121 <quote>tag</quote> a patch with a <emphasis>guard</emphasis>,
122 which is simply a text string of your choosing, then tell MQ to
123 select specific guards to use when applying patches. MQ will
124 then either apply, or skip over, a guarded patch, depending on
125 the guards that you have selected.</para>
127 <para id="x_16e">A patch can have an arbitrary number of guards; each one is
128 <emphasis>positive</emphasis> (<quote>apply this patch if this
129 guard is selected</quote>) or <emphasis>negative</emphasis>
130 (<quote>skip this patch if this guard is selected</quote>). A
131 patch with no guards is always applied.</para>
133 </sect1>
134 <sect1>
135 <title>Controlling the guards on a patch</title>
137 <para id="x_16f">The <command role="hg-ext-mq">qguard</command> command lets
138 you determine which guards should apply to a patch, or display
139 the guards that are already in effect. Without any arguments, it
140 displays the guards on the current topmost patch.</para>
142 &interaction.mq.guards.qguard;
144 <para id="x_170">To set a positive guard on a patch, prefix the name of the
145 guard with a <quote><literal>+</literal></quote>.</para>
147 &interaction.mq.guards.qguard.pos;
149 <para id="x_171">To set a negative guard
150 on a patch, prefix the name of the guard with a
151 <quote><literal>-</literal></quote>.</para>
153 &interaction.mq.guards.qguard.neg;
155 <para id="x_74a">Notice that we prefixed the arguments to the <command>hg
156 qguard</command> command with a <literal>--</literal> here, so
157 that Mercurial would not interpret the text
158 <literal>-quux</literal> as an option.</para>
160 <note>
161 <title>Setting vs. modifying</title>
163 <para id="x_172"> The <command role="hg-ext-mq">qguard</command> command
164 <emphasis>sets</emphasis> the guards on a patch; it doesn't
165 <emphasis>modify</emphasis> them. What this means is that if
166 you run <command role="hg-cmd">hg qguard +a +b</command> on a
167 patch, then <command role="hg-cmd">hg qguard +c</command> on
168 the same patch, the <emphasis>only</emphasis> guard that will
169 be set on it afterwards is <literal>+c</literal>.</para>
170 </note>
172 <para id="x_173">Mercurial stores guards in the <filename
173 role="special">series</filename> file; the form in which they
174 are stored is easy both to understand and to edit by hand. (In
175 other words, you don't have to use the <command
176 role="hg-ext-mq">qguard</command> command if you don't want
177 to; it's okay to simply edit the <filename
178 role="special">series</filename> file.)</para>
180 &interaction.mq.guards.series;
182 </sect1>
183 <sect1>
184 <title>Selecting the guards to use</title>
186 <para id="x_174">The <command role="hg-ext-mq">qselect</command> command
187 determines which guards are active at a given time. The effect
188 of this is to determine which patches MQ will apply the next
189 time you run <command role="hg-ext-mq">qpush</command>. It has
190 no other effect; in particular, it doesn't do anything to
191 patches that are already applied.</para>
193 <para id="x_175">With no arguments, the <command
194 role="hg-ext-mq">qselect</command> command lists the guards
195 currently in effect, one per line of output. Each argument is
196 treated as the name of a guard to apply.</para>
198 &interaction.mq.guards.qselect.foo;
200 <para id="x_176">In case you're interested, the currently selected guards are
201 stored in the <filename role="special">guards</filename> file.</para>
203 &interaction.mq.guards.qselect.cat;
205 <para id="x_177">We can see the effect the selected guards have when we run
206 <command role="hg-ext-mq">qpush</command>.</para>
208 &interaction.mq.guards.qselect.qpush;
210 <para id="x_178">A guard cannot start with a
211 <quote><literal>+</literal></quote> or
212 <quote><literal>-</literal></quote> character. The name of a
213 guard must not contain white space, but most other characters
214 are acceptable. If you try to use a guard with an invalid name,
215 MQ will complain:</para>
217 &interaction.mq.guards.qselect.error;
219 <para id="x_179">Changing the selected guards changes the patches that are
220 applied.</para>
222 &interaction.mq.guards.qselect.quux;
224 <para id="x_17a">You can see in the example below that negative guards take
225 precedence over positive guards.</para>
227 &interaction.mq.guards.qselect.foobar;
229 </sect1>
230 <sect1>
231 <title>MQ's rules for applying patches</title>
233 <para id="x_17b">The rules that MQ uses when deciding whether to apply a
234 patch are as follows.</para>
235 <itemizedlist>
236 <listitem><para id="x_17c">A patch that has no guards is always
237 applied.</para>
238 </listitem>
239 <listitem><para id="x_17d">If the patch has any negative guard that matches
240 any currently selected guard, the patch is skipped.</para>
241 </listitem>
242 <listitem><para id="x_17e">If the patch has any positive guard that matches
243 any currently selected guard, the patch is applied.</para>
244 </listitem>
245 <listitem><para id="x_17f">If the patch has positive or negative guards,
246 but none matches any currently selected guard, the patch is
247 skipped.</para>
248 </listitem></itemizedlist>
250 </sect1>
251 <sect1>
252 <title>Trimming the work environment</title>
254 <para id="x_180">In working on the device driver I mentioned earlier, I don't
255 apply the patches to a normal Linux kernel tree. Instead, I use
256 a repository that contains only a snapshot of the source files
257 and headers that are relevant to Infiniband development. This
258 repository is 1% the size of a kernel repository, so it's easier
259 to work with.</para>
261 <para id="x_181">I then choose a <quote>base</quote> version on top of which
262 the patches are applied. This is a snapshot of the Linux kernel
263 tree as of a revision of my choosing. When I take the snapshot,
264 I record the changeset ID from the kernel repository in the
265 commit message. Since the snapshot preserves the
266 <quote>shape</quote> and content of the relevant parts of the
267 kernel tree, I can apply my patches on top of either my tiny
268 repository or a normal kernel tree.</para>
270 <para id="x_182">Normally, the base tree atop which the patches apply should
271 be a snapshot of a very recent upstream tree. This best
272 facilitates the development of patches that can easily be
273 submitted upstream with few or no modifications.</para>
275 </sect1>
276 <sect1>
277 <title>Dividing up the <filename role="special">series</filename>
278 file</title>
280 <para id="x_183">I categorise the patches in the <filename
281 role="special">series</filename> file into a number of logical
282 groups. Each section of like patches begins with a block of
283 comments that describes the purpose of the patches that
284 follow.</para>
286 <para id="x_184">The sequence of patch groups that I maintain follows. The
287 ordering of these groups is important; I'll describe why after I
288 introduce the groups.</para>
289 <itemizedlist>
290 <listitem><para id="x_185">The <quote>accepted</quote> group. Patches that
291 the development team has submitted to the maintainer of the
292 Infiniband subsystem, and which he has accepted, but which
293 are not present in the snapshot that the tiny repository is
294 based on. These are <quote>read only</quote> patches,
295 present only to transform the tree into a similar state as
296 it is in the upstream maintainer's repository.</para>
297 </listitem>
298 <listitem><para id="x_186">The <quote>rework</quote> group. Patches that I
299 have submitted, but that the upstream maintainer has
300 requested modifications to before he will accept
301 them.</para>
302 </listitem>
303 <listitem><para id="x_187">The <quote>pending</quote> group. Patches that
304 I have not yet submitted to the upstream maintainer, but
305 which we have finished working on. These will be <quote>read
306 only</quote> for a while. If the upstream maintainer
307 accepts them upon submission, I'll move them to the end of
308 the <quote>accepted</quote> group. If he requests that I
309 modify any, I'll move them to the beginning of the
310 <quote>rework</quote> group.</para>
311 </listitem>
312 <listitem><para id="x_188">The <quote>in progress</quote> group. Patches
313 that are actively being developed, and should not be
314 submitted anywhere yet.</para>
315 </listitem>
316 <listitem><para id="x_189">The <quote>backport</quote> group. Patches that
317 adapt the source tree to older versions of the kernel
318 tree.</para>
319 </listitem>
320 <listitem><para id="x_18a">The <quote>do not ship</quote> group. Patches
321 that for some reason should never be submitted upstream.
322 For example, one such patch might change embedded driver
323 identification strings to make it easier to distinguish, in
324 the field, between an out-of-tree version of the driver and
325 a version shipped by a distribution vendor.</para>
326 </listitem></itemizedlist>
328 <para id="x_18b">Now to return to the reasons for ordering groups of patches
329 in this way. We would like the lowest patches in the stack to
330 be as stable as possible, so that we will not need to rework
331 higher patches due to changes in context. Putting patches that
332 will never be changed first in the <filename
333 role="special">series</filename> file serves this
334 purpose.</para>
336 <para id="x_18c">We would also like the patches that we know we'll need to
337 modify to be applied on top of a source tree that resembles the
338 upstream tree as closely as possible. This is why we keep
339 accepted patches around for a while.</para>
341 <para id="x_18d">The <quote>backport</quote> and <quote>do not ship</quote>
342 patches float at the end of the <filename
343 role="special">series</filename> file. The backport patches
344 must be applied on top of all other patches, and the <quote>do
345 not ship</quote> patches might as well stay out of harm's
346 way.</para>
348 </sect1>
349 <sect1>
350 <title>Maintaining the patch series</title>
352 <para id="x_18e">In my work, I use a number of guards to control which
353 patches are to be applied.</para>
355 <itemizedlist>
356 <listitem><para id="x_18f"><quote>Accepted</quote> patches are guarded with
357 <literal>accepted</literal>. I enable this guard most of
358 the time. When I'm applying the patches on top of a tree
359 where the patches are already present, I can turn this patch
360 off, and the patches that follow it will apply
361 cleanly.</para>
362 </listitem>
363 <listitem><para id="x_190">Patches that are <quote>finished</quote>, but
364 not yet submitted, have no guards. If I'm applying the
365 patch stack to a copy of the upstream tree, I don't need to
366 enable any guards in order to get a reasonably safe source
367 tree.</para>
368 </listitem>
369 <listitem><para id="x_191">Those patches that need reworking before being
370 resubmitted are guarded with
371 <literal>rework</literal>.</para>
372 </listitem>
373 <listitem><para id="x_192">For those patches that are still under
374 development, I use <literal>devel</literal>.</para>
375 </listitem>
376 <listitem><para id="x_193">A backport patch may have several guards, one
377 for each version of the kernel to which it applies. For
378 example, a patch that backports a piece of code to 2.6.9
379 will have a <literal>2.6.9</literal> guard.</para>
380 </listitem></itemizedlist>
381 <para id="x_194">This variety of guards gives me considerable flexibility in
382 determining what kind of source tree I want to end up with. For
383 most situations, the selection of appropriate guards is
384 automated during the build process, but I can manually tune the
385 guards to use for less common circumstances.</para>
387 <sect2>
388 <title>The art of writing backport patches</title>
390 <para id="x_195">Using MQ, writing a backport patch is a simple process.
391 All such a patch has to do is modify a piece of code that uses
392 a kernel feature not present in the older version of the
393 kernel, so that the driver continues to work correctly under
394 that older version.</para>
396 <para id="x_196">A useful goal when writing a good backport patch is to
397 make your code look as if it was written for the older version
398 of the kernel you're targeting. The less obtrusive the patch,
399 the easier it will be to understand and maintain. If you're
400 writing a collection of backport patches to avoid the
401 <quote>rat's nest</quote> effect of lots of
402 <literal>#ifdef</literal>s (hunks of source code that are only
403 used conditionally) in your code, don't introduce
404 version-dependent <literal>#ifdef</literal>s into the patches.
405 Instead, write several patches, each of which makes
406 unconditional changes, and control their application using
407 guards.</para>
409 <para id="x_197">There are two reasons to divide backport patches into a
410 distinct group, away from the <quote>regular</quote> patches
411 whose effects they modify. The first is that intermingling the
412 two makes it more difficult to use a tool like the <literal
413 role="hg-ext">patchbomb</literal> extension to automate the
414 process of submitting the patches to an upstream maintainer.
415 The second is that a backport patch could perturb the context
416 in which a subsequent regular patch is applied, making it
417 impossible to apply the regular patch cleanly
418 <emphasis>without</emphasis> the earlier backport patch
419 already being applied.</para>
421 </sect2>
422 </sect1>
423 <sect1>
424 <title>Useful tips for developing with MQ</title>
426 <sect2>
427 <title>Organising patches in directories</title>
429 <para id="x_198">If you're working on a substantial project with MQ, it's
430 not difficult to accumulate a large number of patches. For
431 example, I have one patch repository that contains over 250
432 patches.</para>
434 <para id="x_199">If you can group these patches into separate logical
435 categories, you can if you like store them in different
436 directories; MQ has no problems with patch names that contain
437 path separators.</para>
439 </sect2>
440 <sect2 id="mq-collab:tips:interdiff">
441 <title>Viewing the history of a patch</title>
443 <para id="x_19a">If you're developing a set of patches over a long time,
444 it's a good idea to maintain them in a repository, as
445 discussed in <xref linkend="sec:mq:repo"/>. If you do
446 so, you'll quickly
447 discover that using the <command role="hg-cmd">hg
448 diff</command> command to look at the history of changes to
449 a patch is unworkable. This is in part because you're looking
450 at the second derivative of the real code (a diff of a diff),
451 but also because MQ adds noise to the process by modifying
452 time stamps and directory names when it updates a
453 patch.</para>
455 <para id="x_19b">However, you can use the <literal
456 role="hg-ext">extdiff</literal> extension, which is bundled
457 with Mercurial, to turn a diff of two versions of a patch into
458 something readable. To do this, you will need a third-party
459 package called <literal role="package">patchutils</literal>
460 <citation>web:patchutils</citation>. This provides a command
461 named <command>interdiff</command>, which shows the
462 differences between two diffs as a diff. Used on two versions
463 of the same diff, it generates a diff that represents the diff
464 from the first to the second version.</para>
466 <para id="x_19c">You can enable the <literal
467 role="hg-ext">extdiff</literal> extension in the usual way,
468 by adding a line to the <literal
469 role="rc-extensions">extensions</literal> section of your
470 <filename role="special">~/.hgrc</filename>.</para>
471 <programlisting>[extensions]
472 extdiff =</programlisting>
473 <para id="x_19d">The <command>interdiff</command> command expects to be
474 passed the names of two files, but the <literal
475 role="hg-ext">extdiff</literal> extension passes the program
476 it runs a pair of directories, each of which can contain an
477 arbitrary number of files. We thus need a small program that
478 will run <command>interdiff</command> on each pair of files in
479 these two directories. This program is available as <filename
480 role="special">hg-interdiff</filename> in the <filename
481 class="directory">examples</filename> directory of the
482 source code repository that accompanies this book. <!--
483 &example.hg-interdiff; --></para>
485 <para id="x_19e">With the <filename role="special">hg-interdiff</filename>
486 program in your shell's search path, you can run it as
487 follows, from inside an MQ patch directory:</para>
488 <programlisting>hg extdiff -p hg-interdiff -r A:B my-change.patch</programlisting>
489 <para id="x_19f">Since you'll probably want to use this long-winded command
490 a lot, you can get <literal role="hg-ext">hgext</literal> to
491 make it available as a normal Mercurial command, again by
492 editing your <filename
493 role="special">~/.hgrc</filename>.</para>
494 <programlisting>[extdiff]
495 cmd.interdiff = hg-interdiff</programlisting>
496 <para id="x_1a0">This directs <literal role="hg-ext">hgext</literal> to
497 make an <literal>interdiff</literal> command available, so you
498 can now shorten the previous invocation of <command
499 role="hg-ext-extdiff">extdiff</command> to something a
500 little more wieldy.</para>
501 <programlisting>hg interdiff -r A:B my-change.patch</programlisting>
503 <note>
504 <para id="x_1a1"> The <command>interdiff</command> command works well
505 only if the underlying files against which versions of a
506 patch are generated remain the same. If you create a patch,
507 modify the underlying files, and then regenerate the patch,
508 <command>interdiff</command> may not produce useful
509 output.</para>
510 </note>
512 <para id="x_1a2">The <literal role="hg-ext">extdiff</literal> extension is
513 useful for more than merely improving the presentation of MQ
514 patches. To read more about it, go to <xref
515 linkend="sec:hgext:extdiff"/>.</para>
517 </sect2>
518 </sect1>
519 </chapter>
521 <!--
522 local variables:
523 sgml-parent-document: ("00book.xml" "book" "chapter")
524 end:
525 -->