[Distutils] Current Python packaging status (from my point of view)

[Nick Coghlan]

On 1 November 2016 at 20:22, Paul Moore <p.f.moore at gmail.com> wrote:
> On 1 November 2016 at 09:50, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> There's also a 4th variant, which is getting to a point where someone
>> figures out a pushbutton solution for a build pipeline in a public
>> PaaS that offers a decent free tier.
>
> This, of course, is relatively easy for the simple cases (I could set
> up a project that builds Windows wheels on Appveyor relatively easily
> - I may well do so when I have some spare time). But the "simple
> cases" are precisely those where the only difficulty is the
> (relatively easy to overcome) need for a compiler. In other words, if
> "pip wheel foo" works when you have a compiler, it's simple to
> automate the building of wheels.

It isn't that simple, as what you really want to automate is the
*release process*, where you upload an sdist, and the wheels *just
happen* for:

- the Python versions you want to support (e.g 2.7, 3.4, 3.5)
- the platforms you want to support (typically x86_64 for Windows, Mac
OS X, manylinux1, and maybe 32 bit Windows as well)

Adding a new Python release or a new platform to the build
configuration is currently an activity that requires per-project work
when in theory a build service could just add it automatically based
on when new releases happen.

> The significant problem here is the process of setting up build
> dependencies for more complex projects, like scipy, pywin32, or lxml,
> or even simpler cases such as pyyaml. Each such project would need
> custom setup steps, and there's no easy way for a generic wheel
> building service to infer those steps - the onus would necessarily be
> on the individual projects to provide a script that manages the build
> process (and such a script might as well be the setup,py, meaning that
> we are back to the "pip wheel foo just works" case).

Making "pip wheel foo" work cross platform and defining the steps
needed to cut an sdist for each release are the only pieces that
should genuinely require work on a per-project basis. However, at the
moment, there are a lot of other steps related to the per-release
build-and-publish matrix that are also defined on a per-project basis,
and there's no fundamental reason that they need to be - it's purely a
matter of providing a public build service and keeping it running
smoothly isn't something you want to be asking volunteers to be
responsible for.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia