1.1 --- a/es/mq-collab.tex	Sat Oct 18 07:48:21 2008 -0500
     1.2 +++ b/es/mq-collab.tex	Thu Nov 06 23:09:47 2008 -0500
     1.3 @@ -0,0 +1,393 @@
     1.4 +\chapter{Advanced uses of Mercurial Queues}
     1.5 +\label{chap:mq-collab}
     1.6 +
     1.7 +While it's easy to pick up straightforward uses of Mercurial Queues,
     1.8 +use of a little discipline and some of MQ's less frequently used
     1.9 +capabilities makes it possible to work in complicated development
    1.10 +environments.
    1.11 +
    1.12 +In this chapter, I will use as an example a technique I have used to
    1.13 +manage the development of an Infiniband device driver for the Linux
    1.14 +kernel.  The driver in question is large (at least as drivers go),
    1.15 +with 25,000 lines of code spread across 35 source files.  It is
    1.16 +maintained by a small team of developers.
    1.17 +
    1.18 +While much of the material in this chapter is specific to Linux, the
    1.19 +same principles apply to any code base for which you're not the
    1.20 +primary owner, and upon which you need to do a lot of development.
    1.21 +
    1.22 +\section{The problem of many targets}
    1.23 +
    1.24 +The Linux kernel changes rapidly, and has never been internally
    1.25 +stable; developers frequently make drastic changes between releases.
    1.26 +This means that a version of the driver that works well with a
    1.27 +particular released version of the kernel will not even \emph{compile}
    1.28 +correctly against, typically, any other version.
    1.29 +
    1.30 +To maintain a driver, we have to keep a number of distinct versions of
    1.31 +Linux in mind.
    1.32 +\begin{itemize}
    1.33 +\item One target is the main Linux kernel development tree.
    1.34 +  Maintenance of the code is in this case partly shared by other
    1.35 +  developers in the kernel community, who make ``drive-by''
    1.36 +  modifications to the driver as they develop and refine kernel
    1.37 +  subsystems.
    1.38 +\item We also maintain a number of ``backports'' to older versions of
    1.39 +  the Linux kernel, to support the needs of customers who are running
    1.40 +  older Linux distributions that do not incorporate our drivers.  (To
    1.41 +  \emph{backport} a piece of code is to modify it to work in an older
    1.42 +  version of its target environment than the version it was developed
    1.43 +  for.)
    1.44 +\item Finally, we make software releases on a schedule that is
    1.45 +  necessarily not aligned with those used by Linux distributors and
    1.46 +  kernel developers, so that we can deliver new features to customers
    1.47 +  without forcing them to upgrade their entire kernels or
    1.48 +  distributions.
    1.49 +\end{itemize}
    1.50 +
    1.51 +\subsection{Tempting approaches that don't work well}
    1.52 +
    1.53 +There are two ``standard'' ways to maintain a piece of software that
    1.54 +has to target many different environments.
    1.55 +
    1.56 +The first is to maintain a number of branches, each intended for a
    1.57 +single target.  The trouble with this approach is that you must
    1.58 +maintain iron discipline in the flow of changes between repositories.
    1.59 +A new feature or bug fix must start life in a ``pristine'' repository,
    1.60 +then percolate out to every backport repository.  Backport changes are
    1.61 +more limited in the branches they should propagate to; a backport
    1.62 +change that is applied to a branch where it doesn't belong will
    1.63 +probably stop the driver from compiling.
    1.64 +
    1.65 +The second is to maintain a single source tree filled with conditional
    1.66 +statements that turn chunks of code on or off depending on the
    1.67 +intended target.  Because these ``ifdefs'' are not allowed in the
    1.68 +Linux kernel tree, a manual or automatic process must be followed to
    1.69 +strip them out and yield a clean tree.  A code base maintained in this
    1.70 +fashion rapidly becomes a rat's nest of conditional blocks that are
    1.71 +difficult to understand and maintain.
    1.72 +
    1.73 +Neither of these approaches is well suited to a situation where you
    1.74 +don't ``own'' the canonical copy of a source tree.  In the case of a
    1.75 +Linux driver that is distributed with the standard kernel, Linus's
    1.76 +tree contains the copy of the code that will be treated by the world
    1.77 +as canonical.  The upstream version of ``my'' driver can be modified
    1.78 +by people I don't know, without me even finding out about it until
    1.79 +after the changes show up in Linus's tree.  
    1.80 +
    1.81 +These approaches have the added weakness of making it difficult to
    1.82 +generate well-formed patches to submit upstream.
    1.83 +
    1.84 +In principle, Mercurial Queues seems like a good candidate to manage a
    1.85 +development scenario such as the above.  While this is indeed the
    1.86 +case, MQ contains a few added features that make the job more
    1.87 +pleasant.
    1.88 +
    1.89 +\section{Conditionally applying patches with 
    1.90 +  guards}
    1.91 +
    1.92 +Perhaps the best way to maintain sanity with so many targets is to be
    1.93 +able to choose specific patches to apply for a given situation.  MQ
    1.94 +provides a feature called ``guards'' (which originates with quilt's
    1.95 +\texttt{guards} command) that does just this.  To start off, let's
    1.96 +create a simple repository for experimenting in.
    1.97 +\interaction{mq.guards.init}
    1.98 +This gives us a tiny repository that contains two patches that don't
    1.99 +have any dependencies on each other, because they touch different files.
   1.100 +
   1.101 +The idea behind conditional application is that you can ``tag'' a
   1.102 +patch with a \emph{guard}, which is simply a text string of your
   1.103 +choosing, then tell MQ to select specific guards to use when applying
   1.104 +patches.  MQ will then either apply, or skip over, a guarded patch,
   1.105 +depending on the guards that you have selected.
   1.106 +
   1.107 +A patch can have an arbitrary number of guards;
   1.108 +each one is \emph{positive} (``apply this patch if this guard is
   1.109 +selected'') or \emph{negative} (``skip this patch if this guard is
   1.110 +selected'').  A patch with no guards is always applied.
   1.111 +
   1.112 +\section{Controlling the guards on a patch}
   1.113 +
   1.114 +The \hgxcmd{mq}{qguard} command lets you determine which guards should
   1.115 +apply to a patch, or display the guards that are already in effect.
   1.116 +Without any arguments, it displays the guards on the current topmost
   1.117 +patch.
   1.118 +\interaction{mq.guards.qguard}
   1.119 +To set a positive guard on a patch, prefix the name of the guard with
   1.120 +a ``\texttt{+}''.
   1.121 +\interaction{mq.guards.qguard.pos}
   1.122 +To set a negative guard on a patch, prefix the name of the guard with
   1.123 +a ``\texttt{-}''.
   1.124 +\interaction{mq.guards.qguard.neg}
   1.125 +
   1.126 +\begin{note}
   1.127 +  The \hgxcmd{mq}{qguard} command \emph{sets} the guards on a patch; it
   1.128 +  doesn't \emph{modify} them.  What this means is that if you run
   1.129 +  \hgcmdargs{qguard}{+a +b} on a patch, then \hgcmdargs{qguard}{+c} on
   1.130 +  the same patch, the \emph{only} guard that will be set on it
   1.131 +  afterwards is \texttt{+c}.
   1.132 +\end{note}
   1.133 +
   1.134 +Mercurial stores guards in the \sfilename{series} file; the form in
   1.135 +which they are stored is easy both to understand and to edit by hand.
   1.136 +(In other words, you don't have to use the \hgxcmd{mq}{qguard} command if
   1.137 +you don't want to; it's okay to simply edit the \sfilename{series}
   1.138 +file.)
   1.139 +\interaction{mq.guards.series}
   1.140 +
   1.141 +\section{Selecting the guards to use}
   1.142 +
   1.143 +The \hgxcmd{mq}{qselect} command determines which guards are active at a
   1.144 +given time.  The effect of this is to determine which patches MQ will
   1.145 +apply the next time you run \hgxcmd{mq}{qpush}.  It has no other effect; in
   1.146 +particular, it doesn't do anything to patches that are already
   1.147 +applied.
   1.148 +
   1.149 +With no arguments, the \hgxcmd{mq}{qselect} command lists the guards
   1.150 +currently in effect, one per line of output.  Each argument is treated
   1.151 +as the name of a guard to apply.
   1.152 +\interaction{mq.guards.qselect.foo}
   1.153 +In case you're interested, the currently selected guards are stored in
   1.154 +the \sfilename{guards} file.
   1.155 +\interaction{mq.guards.qselect.cat}
   1.156 +We can see the effect the selected guards have when we run
   1.157 +\hgxcmd{mq}{qpush}.
   1.158 +\interaction{mq.guards.qselect.qpush}
   1.159 +
   1.160 +A guard cannot start with a ``\texttt{+}'' or ``\texttt{-}''
   1.161 +character.  The name of a guard must not contain white space, but most
   1.162 +other characters are acceptable.  If you try to use a guard with an
   1.163 +invalid name, MQ will complain:
   1.164 +\interaction{mq.guards.qselect.error} 
   1.165 +Changing the selected guards changes the patches that are applied.
   1.166 +\interaction{mq.guards.qselect.quux} 
   1.167 +You can see in the example below that negative guards take precedence
   1.168 +over positive guards.
   1.169 +\interaction{mq.guards.qselect.foobar}
   1.170 +
   1.171 +\section{MQ's rules for applying patches}
   1.172 +
   1.173 +The rules that MQ uses when deciding whether to apply a patch
   1.174 +are as follows.
   1.175 +\begin{itemize}
   1.176 +\item A patch that has no guards is always applied.
   1.177 +\item If the patch has any negative guard that matches any currently
   1.178 +  selected guard, the patch is skipped.
   1.179 +\item If the patch has any positive guard that matches any currently
   1.180 +  selected guard, the patch is applied.
   1.181 +\item If the patch has positive or negative guards, but none matches
   1.182 +  any currently selected guard, the patch is skipped.
   1.183 +\end{itemize}
   1.184 +
   1.185 +\section{Trimming the work environment}
   1.186 +
   1.187 +In working on the device driver I mentioned earlier, I don't apply the
   1.188 +patches to a normal Linux kernel tree.  Instead, I use a repository
   1.189 +that contains only a snapshot of the source files and headers that are
   1.190 +relevant to Infiniband development.  This repository is~1\% the size
   1.191 +of a kernel repository, so it's easier to work with.
   1.192 +
   1.193 +I then choose a ``base'' version on top of which the patches are
   1.194 +applied.  This is a snapshot of the Linux kernel tree as of a revision
   1.195 +of my choosing.  When I take the snapshot, I record the changeset ID
   1.196 +from the kernel repository in the commit message.  Since the snapshot
   1.197 +preserves the ``shape'' and content of the relevant parts of the
   1.198 +kernel tree, I can apply my patches on top of either my tiny
   1.199 +repository or a normal kernel tree.
   1.200 +
   1.201 +Normally, the base tree atop which the patches apply should be a
   1.202 +snapshot of a very recent upstream tree.  This best facilitates the
   1.203 +development of patches that can easily be submitted upstream with few
   1.204 +or no modifications.
   1.205 +
   1.206 +\section{Dividing up the \sfilename{series} file}
   1.207 +
   1.208 +I categorise the patches in the \sfilename{series} file into a number
   1.209 +of logical groups.  Each section of like patches begins with a block
   1.210 +of comments that describes the purpose of the patches that follow.
   1.211 +
   1.212 +The sequence of patch groups that I maintain follows.  The ordering of
   1.213 +these groups is important; I'll describe why after I introduce the
   1.214 +groups.
   1.215 +\begin{itemize}
   1.216 +\item The ``accepted'' group.  Patches that the development team has
   1.217 +  submitted to the maintainer of the Infiniband subsystem, and which
   1.218 +  he has accepted, but which are not present in the snapshot that the
   1.219 +  tiny repository is based on.  These are ``read only'' patches,
   1.220 +  present only to transform the tree into a similar state as it is in
   1.221 +  the upstream maintainer's repository.
   1.222 +\item The ``rework'' group.  Patches that I have submitted, but that
   1.223 +  the upstream maintainer has requested modifications to before he
   1.224 +  will accept them.
   1.225 +\item The ``pending'' group.  Patches that I have not yet submitted to
   1.226 +  the upstream maintainer, but which we have finished working on.
   1.227 +  These will be ``read only'' for a while.  If the upstream maintainer
   1.228 +  accepts them upon submission, I'll move them to the end of the
   1.229 +  ``accepted'' group.  If he requests that I modify any, I'll move
   1.230 +  them to the beginning of the ``rework'' group.
   1.231 +\item The ``in progress'' group.  Patches that are actively being
   1.232 +  developed, and should not be submitted anywhere yet.
   1.233 +\item The ``backport'' group.  Patches that adapt the source tree to
   1.234 +  older versions of the kernel tree.
   1.235 +\item The ``do not ship'' group.  Patches that for some reason should
   1.236 +  never be submitted upstream.  For example, one such patch might
   1.237 +  change embedded driver identification strings to make it easier to
   1.238 +  distinguish, in the field, between an out-of-tree version of the
   1.239 +  driver and a version shipped by a distribution vendor.
   1.240 +\end{itemize}
   1.241 +
   1.242 +Now to return to the reasons for ordering groups of patches in this
   1.243 +way.  We would like the lowest patches in the stack to be as stable as
   1.244 +possible, so that we will not need to rework higher patches due to
   1.245 +changes in context.  Putting patches that will never be changed first
   1.246 +in the \sfilename{series} file serves this purpose.
   1.247 +
   1.248 +We would also like the patches that we know we'll need to modify to be
   1.249 +applied on top of a source tree that resembles the upstream tree as
   1.250 +closely as possible.  This is why we keep accepted patches around for
   1.251 +a while.
   1.252 +
   1.253 +The ``backport'' and ``do not ship'' patches float at the end of the
   1.254 +\sfilename{series} file.  The backport patches must be applied on top
   1.255 +of all other patches, and the ``do not ship'' patches might as well
   1.256 +stay out of harm's way.
   1.257 +
   1.258 +\section{Maintaining the patch series}
   1.259 +
   1.260 +In my work, I use a number of guards to control which patches are to
   1.261 +be applied.
   1.262 +
   1.263 +\begin{itemize}
   1.264 +\item ``Accepted'' patches are guarded with \texttt{accepted}.  I
   1.265 +  enable this guard most of the time.  When I'm applying the patches
   1.266 +  on top of a tree where the patches are already present, I can turn
   1.267 +  this patch off, and the patches that follow it will apply cleanly.
   1.268 +\item Patches that are ``finished'', but not yet submitted, have no
   1.269 +  guards.  If I'm applying the patch stack to a copy of the upstream
   1.270 +  tree, I don't need to enable any guards in order to get a reasonably
   1.271 +  safe source tree.
   1.272 +\item Those patches that need reworking before being resubmitted are
   1.273 +  guarded with \texttt{rework}.
   1.274 +\item For those patches that are still under development, I use
   1.275 +  \texttt{devel}.
   1.276 +\item A backport patch may have several guards, one for each version
   1.277 +  of the kernel to which it applies.  For example, a patch that
   1.278 +  backports a piece of code to~2.6.9 will have a~\texttt{2.6.9} guard.
   1.279 +\end{itemize}
   1.280 +This variety of guards gives me considerable flexibility in
   1.281 +qdetermining what kind of source tree I want to end up with.  For most
   1.282 +situations, the selection of appropriate guards is automated during
   1.283 +the build process, but I can manually tune the guards to use for less
   1.284 +common circumstances.
   1.285 +
   1.286 +\subsection{The art of writing backport patches}
   1.287 +
   1.288 +Using MQ, writing a backport patch is a simple process.  All such a
   1.289 +patch has to do is modify a piece of code that uses a kernel feature
   1.290 +not present in the older version of the kernel, so that the driver
   1.291 +continues to work correctly under that older version.
   1.292 +
   1.293 +A useful goal when writing a good backport patch is to make your code
   1.294 +look as if it was written for the older version of the kernel you're
   1.295 +targeting.  The less obtrusive the patch, the easier it will be to
   1.296 +understand and maintain.  If you're writing a collection of backport
   1.297 +patches to avoid the ``rat's nest'' effect of lots of
   1.298 +\texttt{\#ifdef}s (hunks of source code that are only used
   1.299 +conditionally) in your code, don't introduce version-dependent
   1.300 +\texttt{\#ifdef}s into the patches.  Instead, write several patches,
   1.301 +each of which makes unconditional changes, and control their
   1.302 +application using guards.
   1.303 +
   1.304 +There are two reasons to divide backport patches into a distinct
   1.305 +group, away from the ``regular'' patches whose effects they modify.
   1.306 +The first is that intermingling the two makes it more difficult to use
   1.307 +a tool like the \hgext{patchbomb} extension to automate the process of
   1.308 +submitting the patches to an upstream maintainer.  The second is that
   1.309 +a backport patch could perturb the context in which a subsequent
   1.310 +regular patch is applied, making it impossible to apply the regular
   1.311 +patch cleanly \emph{without} the earlier backport patch already being
   1.312 +applied.
   1.313 +
   1.314 +\section{Useful tips for developing with MQ}
   1.315 +
   1.316 +\subsection{Organising patches in directories}
   1.317 +
   1.318 +If you're working on a substantial project with MQ, it's not difficult
   1.319 +to accumulate a large number of patches.  For example, I have one
   1.320 +patch repository that contains over 250 patches.
   1.321 +
   1.322 +If you can group these patches into separate logical categories, you
   1.323 +can if you like store them in different directories; MQ has no
   1.324 +problems with patch names that contain path separators.
   1.325 +
   1.326 +\subsection{Viewing the history of a patch}
   1.327 +\label{mq-collab:tips:interdiff}
   1.328 +
   1.329 +If you're developing a set of patches over a long time, it's a good
   1.330 +idea to maintain them in a repository, as discussed in
   1.331 +section~\ref{sec:mq:repo}.  If you do so, you'll quickly discover that
   1.332 +using the \hgcmd{diff} command to look at the history of changes to a
   1.333 +patch is unworkable.  This is in part because you're looking at the
   1.334 +second derivative of the real code (a diff of a diff), but also
   1.335 +because MQ adds noise to the process by modifying time stamps and
   1.336 +directory names when it updates a patch.
   1.337 +
   1.338 +However, you can use the \hgext{extdiff} extension, which is bundled
   1.339 +with Mercurial, to turn a diff of two versions of a patch into
   1.340 +something readable.  To do this, you will need a third-party package
   1.341 +called \package{patchutils}~\cite{web:patchutils}.  This provides a
   1.342 +command named \command{interdiff}, which shows the differences between
   1.343 +two diffs as a diff.  Used on two versions of the same diff, it
   1.344 +generates a diff that represents the diff from the first to the second
   1.345 +version.
   1.346 +
   1.347 +You can enable the \hgext{extdiff} extension in the usual way, by
   1.348 +adding a line to the \rcsection{extensions} section of your \hgrc.
   1.349 +\begin{codesample2}
   1.350 +  [extensions]
   1.351 +  extdiff =
   1.352 +\end{codesample2}
   1.353 +The \command{interdiff} command expects to be passed the names of two
   1.354 +files, but the \hgext{extdiff} extension passes the program it runs a
   1.355 +pair of directories, each of which can contain an arbitrary number of
   1.356 +files.  We thus need a small program that will run \command{interdiff}
   1.357 +on each pair of files in these two directories.  This program is
   1.358 +available as \sfilename{hg-interdiff} in the \dirname{examples}
   1.359 +directory of the source code repository that accompanies this book.
   1.360 +\excode{hg-interdiff}
   1.361 +
   1.362 +With the \sfilename{hg-interdiff} program in your shell's search path,
   1.363 +you can run it as follows, from inside an MQ patch directory:
   1.364 +\begin{codesample2}
   1.365 +  hg extdiff -p hg-interdiff -r A:B my-change.patch
   1.366 +\end{codesample2}
   1.367 +Since you'll probably want to use this long-winded command a lot, you
   1.368 +can get \hgext{hgext} to make it available as a normal Mercurial
   1.369 +command, again by editing your \hgrc.
   1.370 +\begin{codesample2}
   1.371 +  [extdiff]
   1.372 +  cmd.interdiff = hg-interdiff
   1.373 +\end{codesample2}
   1.374 +This directs \hgext{hgext} to make an \texttt{interdiff} command
   1.375 +available, so you can now shorten the previous invocation of
   1.376 +\hgxcmd{extdiff}{extdiff} to something a little more wieldy.
   1.377 +\begin{codesample2}
   1.378 +  hg interdiff -r A:B my-change.patch
   1.379 +\end{codesample2}
   1.380 +
   1.381 +\begin{note}
   1.382 +  The \command{interdiff} command works well only if the underlying
   1.383 +  files against which versions of a patch are generated remain the
   1.384 +  same.  If you create a patch, modify the underlying files, and then
   1.385 +  regenerate the patch, \command{interdiff} may not produce useful
   1.386 +  output.
   1.387 +\end{note}
   1.388 +
   1.389 +The \hgext{extdiff} extension is useful for more than merely improving
   1.390 +the presentation of MQ~patches.  To read more about it, go to
   1.391 +section~\ref{sec:hgext:extdiff}.
   1.392 +
   1.393 +%%% Local Variables: 
   1.394 +%%% mode: latex
   1.395 +%%% TeX-master: "00book"
   1.396 +%%% End: