[Distutils] PEP439 and backward compat / easy_install / distlib

[Daniel Holth]

On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray <erik.m.bray at gmail.com> wrote:
> On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth <dholth at gmail.com> wrote:
>> My vision for the setuptools deprecation process is that distutils
>> rides into the sunset with it. In this future eventually bugs in
>> setuptools will be solved by porting setup.py to one of (X, Y, Z)
>> which haven't necessarily been invented yet.
>
> That would be nice (really!) but what are you proposing replace it for
> building packages with heavy reliance on C extensions?  Because for
> that one use case (and perhaps that alone) it works "pretty okay" for
> most cases.  I don't want to start seeing an infinite number of ways
> to configure and build extension modules.  The great thing about using
> distutils (or some variant) for this is that if I had the source for a
> package I could just `./setup.py build` and it would "just work" for
> all but the most complex cases (SciPy for example).
>
> I don't want to have a situation where some projects are using bento
> and others are using scons and some are using waf and others are using
> autoconf, etc, etc.  It's fine if a few projects have their own
> special needs for build toolchains and I've been saying all along that
> building should be separate from installing, and it should be easier
> to drop in one's own build system.
>
> Another thing that setuptools provides that currently works "pretty
> well" with extension modules is `./setup.py develop`.  It calls
> `setup.py build_ext --inplace` to make extension modules importable.
> Any build system for extension modules needs to be able to do
> something similar to support in-place install functionality like
> `setup.py develop`, `pip install -e`, etc.

It's true that de-standardization of the build process has its own
problems. As a consolation you don't have to do it as often because we
have binary packages. It's hard to say how much fragmentation will
happen, but distutils Extension() is an awful way to compile things!
It has very little to recommend it apart from that it's there. For
example, no parallel builds and no partial re-compiles based on what's
changed. The trouble is that we know the packages on the cheeseshop
mostly work but it's harder to count the stuff that avoids the
cheeseshop because setuptools and setup.py wasn't a good solution.

The example I've had to deal with recently is pycairo. They already
use waf to compile their Python extension and don't use setup.py at
all, so I argue that de-standardization has already happened. Whatever
very easy (for pycairo) thing we can do make them "pip install"-able
again is a plus.

Even my own simple bcrypt wrapper "cryptacular" would compile an
assembler file if I knew how to make that work with distutils. It
doesn't and it's a little slower than it would have been if I had a
decent build tool.

I suspect at least 80% of packages will use some simple thing that
comes with Python, two third party build tools will dominate, and we
will discover interesting things that just weren't possible before. At
least if someone wants to improve packaging we can make it easy for
them to try without having to ask distutils-sig.