[Python-ideas] Wheels For ...

[Donald Stufft]

On September 6, 2015 at 9:27:32 PM, Steven D'Aprano (steve at pearwood.info) wrote:
> On Sun, Sep 06, 2015 at 07:33:29PM +0200, Sven R. Kunze wrote:
>  
> > Why do developers need to build their distribution themselves?
> >
> > I had not real answer to him, but pondering a while over it, I found it
> > really insightful. Viewing this from a different angle, packaging your
> > own distribution is actually a waste of time. It is a tedious,
> > error-prone task involving no creativity whatsoever. Developers on the
> > other hand are actually people with very little time and a lot of
> > creativity at hand which should spend better. The logical conclusion
> > would be that PyPI should build wheels for the developers for every
> > python/platform combination necessary.
>  
> Over on the python-list mailing list, Ned Batchelder asked a question. I
> haven't seen an answer there, and as far as I know he isn't subscribed
> here, so I'll take the liberty of copying his question here:
>  
> Ned says:
>  
> "As a developer of a Python package, I don't see how this would be
> better. The developer would still have to get their software into some
> kind of uniform configuration, so the central authority could package
> it. You've moved the problem from, "everyone has to make wheels" to
> "everyone has to make a tree that's structured properly." But if we can
> do the second thing, the first thing is really easy.
>  
> Maybe I've misunderstood?"
>  
>  


A PyPI build farm for authors is something I plan on getting too if someone
doesn't beat me to it. It won't be mandatory to use it, and it's not going to
cover every corner case where you need some crazy obscure library installed,
but it will ideally try to make things better.

As far why it's better, it's actually pretty simple. Let's take lxml for
example which binds against libxml2. It needs built on Windows, it needs built
on OSX, it needs built on various Linux distributions in order to cover the
spread of just the common cases. If we want to start to get into uncommon
platforms we're looking at various BSDs, Solaris, etc. It's a lot of
infrastructure to maintain for the common cases much less the uncommon cases
that we can centralize the maintenance into one location. In addition to all of
that, it turns out you pretty much need to get most of the way to defining that
configuration in a central location anyways since pip already needs to know how
to build your project, the only things it doesn't know is what platforms you
support and what, if any, extra libraries you require to be installed. There
are some PEPs in the works that may make that second part known ahead of time
and for the first one, we can simply ask when a project enable the build farm.

Even if we only support things that don't require additional libraries to be 
installed, that's still a pretty big win by default that will allow a wide
number of projects to be installed from a binary distribution and not require
a compiler toolchain that previously couldn't be due to authors not willing or
not able to manage the overhead of the infrastructure around building on all
of those platforms.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA