[Python-Dev] Status of packaging in 3.3

[Éric Araujo]

   Hi all,

   We need to make a decision about the packaging module in Python 3.3. 
  Please read this message and breathe deeply before replying :)

   [Sorry this ends up being so long; Tarek, Georg, Guido, I hope you 
have the time to read it.]

   Let me first summarize the history of packaging in the standard 
library.  (Please correct if I misremember something; this email 
expresses my opinion and I did not talk with Tarek or Alexis before 
writing it.)  Two years ago the distutils2 (hereafter d2) project was 
started outside of the stdlib, to allow for fast-paced changes, releases 
and testing before merging it back.  Changes in distutils were reverted 
to go back to misfeature-for-misfeature compatibility (but not 
bug-for-bug: bug fixes are allowed, unless we know for sure everyone is 
depending on them or working around them).

   Tarek’s first hope was to have something that could be included in 
2.7 and 3.2, but these deadlines came too fast.  At one point near the 
start of 2011 (didn’t find the email) there was a discussion with Martin 
about adding support for the stable ABI or parallel builds to distutils, 
in which Tarek and I opposed adding this new feature to distutils as per 
the freeze policy, and Martin declared he was not willing to work 
outside of the standard library.  We (d2 developers and python-dev) then 
quickly agreed that distutils2 would be merged back after the release of 
3.2, which was done.  There was no PEP requested for this addition, 
maybe because this was not a fully new module but an improvement of an 
existing one with real-world-tested features, or maybe just because 
nobody thought about the process.  In retrospect, a PEP would have 
really helped define the scope of the changes and the roadmap for packaging.

   Real life caused contributors to come and go, and the primary 
maintainer (Tarek at first, me since last December) to be at times very 
busy (like me these last three months), with the result that packaging 
is in my opinion just not ready.  Many big and small things need more 
work: the three packaging PEPs implemented in d2 have small flaws or 
missing pieces (I’m not repeating the list here to avoid spawning 
subthreads) that need to be addressed, we’ve started to get feedback 
from users and developers only recently (pip and buildout devs since 
last PyCon for example) the public Python API of d2 is far from great, 
the implementation is of very unequal quality, important features have 
patches that are not fully ready (—and I do acknowledge that I am the 
blocker for reviews on many of them—), the compiler system has not been 
revised, tests are not all clear and robust, some of the bdist commands 
need to be removed, a new bdist format needs to be designed, etc.

   With beta coming, a way to deal with that unfortunate situation needs 
to be found.  We could (a) grant an exception to packaging to allow 
changes after beta1; (b) keep packaging as it is now under a provisional 
status, with due warnings that many things are expected to change; (c) 
remove the unstable parts and deliver a subset that works (proposed by 
Tarek to the Pyramid author on distutils-sig); (d) not release packaging 
as part of Python 3.3 (I think that was also suggested on distutils-sig 
last month).

   I don’t think (a) would give us enough time; we really want a few 
months (and releases) to hash out the API (most notably with the pip and 
buildout developers) and clean the bdist situation.  Likewise (c) would 
require developer (my) time that is currently in short supply.  (b) also 
requires time and would make development harder, not to mention probable 
user pain.  This leaves (d), after long reflection, as my preferred 
choice, even though I disliked the idea at first (and I fully expect 
Tarek to feel the same way).

   I’d like to stress that this is not as bad as it appears at first. 
We (I) will have to craft reassuring wording to explain why 3.3b1 does 
not include packaging any more, but I believe that it would be worse for 
our users (code-wise and PR-wise) to deliver a half-finished version in 
3.3 rather than to retract it and wait for 3.4.  And if we keep in mind 
that many people are still using a 2.x version, releasing in 3.3 or 3.4 
makes no change for them: the standalone releases on PyPI will keep coming.

   Developer-wise, this would *not* mean that the considerable effort 
that went into porting and merging, and the really appreciated patches 
from developers such as Vinay, would have been in vain: even if 
packaging is removed from the main repo (or just from the release 
systems), there would be a clone to continue development, or the module 
would be added back right after the 3.3 release, or we would develop in 
the d2 repo and merge it back when it’s ready—this is really an 
implementation detail for the decision; my point is that the work will 
not be lost.

   Thanks for reading; please express your opinion (especially Tarek as 
d2 project lead, Georg as RM and Guido as BDFL).