Title: parallelized release management"
DEP: 10
State: DRAFT
Date: 2011-04-30
Drivers: Sean Finney <seanius@debian.org>,
 Raphaƫl Hertzog <hertzog@debian.org>
URL: http://dep.debian.net/deps/dep10
Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep10.mdwn
License: GPL
Abstract: Proposal for changes to release management methodology and 
 infrastructure, allowing the Debian release process to function
 in parallel to non-release related updates.
  1. Introduction / Problem scope
  2. Past and present release methods
    1. Frozen (< 2000)
    2. Testing (2000-Present)
  3. Constraints and requirements for any change to the release process
  4. A host of proposals
    1. Branch testing at freeze time (a.k.a frozen v2.0)
    2. Implement an unstable-updates
    3. Implement "Debian PPA's" and advocate their use
    4. Implement a rolling release

Introduction / Problem scope

Currently, as the project nears a new stable release, a freeze is instituted on the testing suite. The freeze is put in place to allow the release team to focus on resolving the remaining Release Critical (RC) bugs for the next stable release, and at the same time to prevent regressions from new uploads. Typically the freeze begins as an advisory "soft freeze", which over time increases in strictness and levels of enforcement.

Unsuprisingly, as the strictness of the freeze increases, there is an inversely proportional decrease in other non-release targeted maintainer activity. Since unstable still is the preferred route for packages to reach the new release during this period, maintainers are highly discouraged and in some cases prevented from doing non-release targeted activities in unstable.

The reduction of such non-release activity is viewed as problematic in this DEP, for some inter-related reasons:

As Debian is well known for taking a "release when it's ready" approach, the freeze periods are generally known to last considerable amounts of time. Consider the last three freezes:

This means, in rough terms, that the when testing thaws, that the Debian project may be starting from a state half a year behind comparable distributions; the project is, in essence, starting from a 6 month standstill.

This standstill, as well as the subsequent rush of "post-thaw" uploads, will introduce further delays to the next release, as release goals will be set relative to this handicapped starting point and a non-trivial amount of developer effort will be spent reconciling problems in the ensuing rush of uploads and transitions.

Past and present release methods

Frozen (< 2000)

Before the introduction of testing, Debian used a simple release process where the unstable suite was snapshotted into a frozen suite. This suite would then be used exclusively for preparing the next stable release, with unstable continuing in parallel.

Before freeze       Freeze           Release

[unstable/sid]--------------------------------------------------------------
                    \
                    [frozen].-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-.-.-.-.-.

[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.[oldstable/R_N-1].-.-.-(EOL)

--------: Normal activity.  Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
\ \ \ \ : Package migration activity.

Use cases with frozen

Users were divided into two sets, those using unstable and those using stable. Very few users would use both, as the two lines of development would quickly diverge from each other.

Benefits with frozen

Problems with frozen

Testing (2000-Present)

The testing suite was introduced in Debian between the release of potato and woody, in the fall of 20001. The goal was to provide a suite that was in a better state for release preparation, by having both automated and manual tools to keep down the level of bugs and general volatility.

As a pleasant and convenient side-effect, the new suite also provided a "slightly less buggy unstable" for developers and end-users, who wanted newer software/features not available in stable, but wanted some level of protection to the relatively unpredictable nature of unstable.

Before release         Freeze              Release

[unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
     \ \ \ \ \ \ \ \ \  \   \    \       \   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
                                      / / \
                                     / /   [stable/R_N].-.-.-.-.-.-.-.-.-.-.
                                    / /          /          /           /
                               [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-

[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)

--------: Normal activity.  Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
\ \ \ \ : Package migration activity.  Spacing of marks is a rough
          indication of frequency.

During the freeze, the testing suite becomes entirely dedicated to the release work. In practice, this also means that unstable is also to some extent frozen from non-release activity for many packages, since it still serves as the main route to testing for new uploads.

Use cases with testing

The introduction of the new suite also introduced a type of continuum in which users now have more flexibility in selecting what to have installed.

Benefits with testing

Problems with testing

(See Introduction)

Constraints and requirements for any change to the release process

Based on feedback on the debian-devel mailing list, this proposal makes a number of assumptions about constraints and requirements for any alteration of the current release process.

  1. Any irreconcilable conflict should side towards release preparation
  2. activity which can not be done in parellel must only be done for release preparation.
  3. any proposal must be no less safe than the current system with regards to human error (uploads to wrong suite, etc, un-unannounced transitions, etc).
  4. Debian stable releases need sufficient test coverage by users
  5. a testing (or testing-like) quarantine is needed for QA purposes
  6. sufficient users should be using this quarantine to give good coverage.
  7. Any parallel work should interfere minimally with release preparations
  8. newer software should not prevent an upload path to the release.
  9. library transitions and similar should not cause unreasonable overhead.
  10. it must still be possible to remove packages from the release.
  11. The upgrade path and archive state must be consistant and reconcilable
  12. packages updated outside of the release should always have higher versions than in the release, even if both were updated.
  13. If the suite contents are to change post-release, the proposal must account for renamed/duplicate/removed packagets.

A host of proposals

During the post-squeeze release feedback on debian-devel, there has been a number of ideas and suggestions for how the freeze of the release process could be minimized, circumvented, or avoided altogether. Some of the more popular and/or controversial alternatives will be discussed and analyzed in the following section.

Branch testing at freeze time (a.k.a frozen v2.0)

At some point during release preparation, somewhere between the "soft" and "hard" freeze points of the current release, the testing suite is split into two new independant suites. The first of these suites is used for continuing the (frozen) next stable release, while the other continues receiving updates via unstable.

Before release         Freeze                 Release

[unstable/sid]--------------------------------------------------------------
          \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------[testing/R_N+1]--------------------------------------
                           \ \ \  \   \    \
                            [R_N].-.-.-.-.-.-.[stable/R_N].-.-.-.-.-.-.-.-.-
                             /    /   /  / / / /         /                /
                       [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-

[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1]-.-(EOL)

--------: Normal activity.  Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
\ \ \ \ : Package migration activity.  Spacing of marks is a rough
          indication of frequency.

This is very similar to the previous frozen release process, though with the key difference that at the point the branch-off occurs, the suite is in a far better condition due to the additional and continuous QA recieved in the unstable->testing process.

As a consequence, work in unstable will quickly diverge from the state of the frozen release, which has to immediate consequences: * The release team will rely on using the corresponding -proposed-updates pseudo-suite for release-targeted uploads, as undesired transitions/updates in unstable will quickly pare off the candidates for direct migration. * The default path (and userbase) of unstable->testing can no longer be relied upon for the same amount of QA in the release preparation process.

Benefits

Problems

Conclusion

Implement an unstable-updates

A new pseudo-suite unstable-updates (or some other creative name left for a later exercise) is established, which functions in a manner similar to the existing experimental pseudo-suite. A key difference from experimental is that this suite is explicitly expected to feed packages to the unstable suite, and thus the contents should be subject to the same quality standards expected of unstable uploads. Outside of the release process, this feeding can be accomplished either by automatic britney migrations or some form of semi-automatic "gating" process requiring ACK's from either the maintainer or the release team.

During freeze, which continues as it does today, the release team disables the migrations from unstable-updates to unstable. The former, in effect, becomes a staging area for updates to the following release. Post-release, the migration is re-activated, possibly with some additional controls allowing the updates to occur in a more orderly fashion if needed.

Before release         Freeze              Release

[unstable-updates]----------------------------------------------------------
     \ \ \ \ \ \ \ \ \      \            \   \   \ \ \ \ \ \ \ \ \ \ \ \ \ \
[unstable/sid]----------.--.--.--.-.-.-.-.----------------------------------
     \ \ \ \ \ \ \ \ \  \   \    \       \   \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
[testing/R_N]----------.-.-.-.-.-.-.-.-.-.-[testing/R_N+1]------------------
                                      / / \
                                     / /   [stable/R_N].-.-.-.-.-.-.-.-.-.-.
                                    / /          /          /           /
                               [R_N p-u].-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-

[stable/R_N-1].-.-.-.-.-.-.-.-.-.-.-.-.-.-[oldstable/R_N-1].-.-.-(EOL)

--------: Normal activity.  Standard rules for uploads and migrations.
.-.-.-.-: Release targeted activity.  Freezes and limited uploads. 
\ \ \ \ : Package migration activity.  Spacing of marks is a rough
          indication of frequency.

Benefits

Problems

Conclusion

Implement "Debian PPA's" and advocate their use

Another alternative which has been brought up at several points in discussion is the use of Personal Package Archives, a.k.a. PPA's, a.k.a. DSR's, as a means to provide packages updates entirely outside of the release process. Individual maintainers, or teams of related projects, could establish and host independant and self-contained repositories for newer packages.

During a release freeze, these repositories are created on an ad-hoc basis to host any updated packages not targeted for release. Users seeking newer software would be directed to add additional APT sources to their package manager configuration to enable the updates.

Additionally, PPA's could serve several other purposes apart from avoiding the testing freeze. For example, Outside of the release proucess, they can continue to host newer packages than are immediately available in unstable, in a manner similar to the existing experimental pseudo-suite.

In a manner similar to the backports.org services, PPA's could be established to serve self-contained sets of backported packages to previous stable releases. For example, newer versions of popular desktop software, or entirely desktop environments, could be hosted in these self-contained release areas.

PPA's could also be used by maintainers and/or the release team for preparation of package transitions outside of unstable, providing an orderly way to update sets of interdependant packages and reducing overall breakage. This would lead to not only a higher quality experience in testing/unstable, but also streamline the process for subsequent releases, which would experience fewer blockages in work due to ongoing transitions.

Benefits

Problems

Conclusion

Implement a rolling release

NOTE: this section needs to be re-worked. We're not talking about implementing rolling, though there are some key points which do need to be discussed in how this implementation would affect and work with rolling. The idea of a Debian rolling release has been discussed with increased amounts of interest and seriousness during and after the squeeze release cycle.

While it has been proposed in several shapes and forms, the overarching concept is that users should be provided a constantly updating release similar to the current testing suite, with some key differences:

It's worth note that the motivation/proposal for rolling doesn't entirely overlap with the motivations behind the DEP. Certainly both proposals involve the ability to have continuous updates during the entire release cycle, but beyond that the additional desires regarding rolling are orthogonal (or only related in as much as they might place constraints on the complementing implementations)

Furthermore, as stated, there have been several different visions/implementations of rolling discussed.

rolling by way of "alternate entry points"

A new and entirely independant rolling suite is established, in parallel to testing. Both suites have a default entry point of the unstable suite, which feeds both suites through a similar manner.

While packages may migrate into rolling via typical unstable uploads, there also exist means to directly upload to the suite via a rolling-proposed-updates or similarly named pseudo-suite. This pseudo suite would be an overlay on top of the standard rolling suite, allowing for the testing and acceptance of direct fixes without the risk of being blocked by any ongoing transitions or otherwise unsatisfiable dependency chains.

During a release freeze, as the default entry point for rolling,

rolling-proposed-updates could take the additional role of accepting new non-release targeted uploads, or an additional entry point could be defined to replace the role of unstable.

rolling by way of "override hints"

Benefits

Problems

Conclusion