[Python-Dev] Packaging and binary distributions for Python 3.3

[Éric Araujo]

Hi Paul,

Thanks for raising this during the development phase.

> I see that the Packaging documentation is now more complete (at least
> at docs.python.org) - I don't know if it's deemed fully complete yet,
> but I scanned the documentation and "Installing Python Projects" looks
> pretty much converted (and very good!!), but "Distributing Python
> Projects" still has quite a lot of distutils-related text in, and I
> need to read more deeply to understand if that's because it remains
> unchanged, or if it is still to be updated.
The basic structure is in place (distributing/installing/library
reference), but the docs are far from up-to-date.  I have nearly
finished a first patch that fixes a ton of markup issues and updates
distutils idioms (setup.py snippets e.g.) to packaging ones (setup.cfg
snippets, using pysetup, etc.), it’s already a thousand changed lines.
Then I will work on another patch to move things around, consolidate,
expand and rephrase.  See http://bugs.python.org/issue12779 if you want
to follow along and review patches.

> But one thing struck me - the "Installing Python Projects" document
> talks about source distributions, but not much about binary
> distributions.
This is inherited from distutils docs, not a deliberate choice.  We just
haven’t thought much, if at all, about binary distributions.

> On Windows, binary distributions are significantly more important than
> on Unix, because not all users have easy access to a compiler, and
> more importantly, C library dependencies can be difficult to build,
> hard to set up, and generally a pain to deal with.
Are there that many distributions with extension modules?  sdists should
work well even on Windows for pure Python projects.

> I don't know what format packaging favours.
As a direct distutils descendant, can create bdist_wininst and
bdist_msi.  For installing, I was not aware of the problem you reported
(“does not interact well with pysetup”); can you give more info?  I’m
guessing it boils down to the fact that Windows binary installers are
meant to be clicked by users, not managed with command-line tools.

IIRC the current behavior in pysetup is to favor source distributions,
but bdists should probably be favored for non-pure distributions on Windows.

> So there will be a need for a pysetup-friendly binary format.
> I assume that the egg format will fill this role - or is that not the
> case? What is the current thinking on binary distribution formats for
> Python 3.3?
First, we don’t want to include wholesale support for setuptools eggs in
packaging.  We have options to support egg/egg-info metadata in the PEP
376 implementation (packaging.database), because we need that to provide
a useful tool for users and help them switch, but eggs are another
matter.  After all, if setuptools and then pkg_resources were turned
down for inclusion in Python 2.5, it’s not now that we have packaging
that we’ll change our mind and just bless eggs.  What we can do however
is to see what bdist_egg does and define a new bdist command inspired by
it, but without zipping, pkg_resource calls, etc.

> The main reason I am asking is that I would like to write an article
> (or maybe a series of articles) for Python Insider, introducing the
> new packaging facilities from the point of view of an end user with
> straightforward needs (whether a package user just looking to manage a
> set of installed packages, or a module author who just wants to
> publish his code in a form that satisfies as many people as possible).
That’s excellent.  I too thought about writing something about packaging
for that blog, but an outside end-user viewpoint like yours would best
match the readership.  I can write a shorter piece just for packaging
tool developers (i.e. how to use packaging as a library), or you can
write that one too and act as a tester for our doc and API.

> What I'd hope to do is, as well as showing people all the nice things
> they can expect to see in Python 3.3, to also start package authors
> thinking about what they need to do to support their users under the
> new system.
Yes!  We need feedback to provide a much better tool than distutils,
before the API is locked by backward compatibility rules.

I actually wanted to talk about that, so let me take the opportunity.
What if we released packaging in Python 3.3 (and distutils2 1.0 on PyPI)
as a not-quite-final release?  (Something like Python 3.0, which was not
considered a real version and not supported as much as the other ones.)
 The goal would be to expose it to a large range of users to get bug
reports and feature requests, but without locking us forever into one
API or design, which was the death of distutils a year ago.  The idea is
not to scare people with warnings that we’ll break APIs on a whim, but
that we keep the option to change parts of packaging and release a 2.0
with Python 3.4, with documented changes from 3.3.  Opinions?

Regards