[Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 26 May 2016 at 19:20, Matthew Brett <matthew.brett at gmail.com> wrote:
> I just wanted to make sure that we didn't lose out on starting a
> discussion of this problem.
>
> The problem is of course caused by the runaway success of the wheel
> format, and I'm sure it can be solved in a sensible way, but however
> expressed, it's true that wheels have become so standard that we do
> need to think about automation for build and release, if we aren't
> going run into trouble.   By trouble, I mean that users will often hit
> the situation where they don't get wheels when they expect to, and get
> turned off pypi / wheels as a result.  I have personally put a great
> deal of work into building and releasing wheels, so that is something
> I'd really like to avoid.
>
> So - can I humbly ask - what is the best way to get that discussion going?

I would certainly like to promote projects using existing CI systems
for building wheels. It seems like a pretty simple approach that would
cover a decent proportion of the problem.

There's already a section in the Packaging User Guide explaining how
to use Appveyor to build Windows binary wheels. If you don't have
dependencies to deal with (e.g., you're simply providing a C speedup
module) it's pretty simple. If you *do* have dependencies to deal
with, I suspect you could arrange for them to be uploaded when you
build somehow without too much issue - but I don't have projects like
that so I can't really add that to the document. The biggest problem
with Appveyor is that it's less well known than Travis. However, I
don't know of many (frankly, any!) projects using this approach -
whether that's because of a lack of awareness, or because it's not as
useful as it seemed to me when I wrote it, or some other reason, I
don't honestly know. Maybe we could get some feedback on what can be
done to improve that section? Is there going to be any sort of
packaging discussions at PyCon? Maybe it's a question that could be
asked? (I'm not going to be there myself, unfortunately).

For Linux/Unix, the problem was always the plethora of binary formats.
If we can assume that manylinux builds will constitute a 90% solution
here, maybe someone could contribute a section explaining how to add
the building of manylinux wheels into a project's Travis config. I
know almost nothing about OSX, but doesn't Travis offer OSX builders?
Could they be used to build extensions in the same way as the above?

Going beyond this level (which is basically making sure projects know
they have easy access to build environments for the key platforms, and
encouraging them to use them) is when it gets harder. Integrating
artifacts from travis/appveyor into a release process would need some
work, for example. Maybe a way forward here would be for some projects
to try doing so, and writing up a set of "how we did it" notes that
could over time turn into a set of recommended practices in the PUG.
Or an interested 3rd party could probably do most of the
experimentation and development, simply by forking a few typical
projects and seeing what was needed.

With all of the above, we'd end up with some (hopefully!) pretty good
documentation of how to (relatively) painlessly add building of wheels
for all major platforms into *any* project (that didn't have
specialised issues to deal with).

I'm not sure where we'd go from there. The next step seems to be to
actually provide some sort of build service, or curated set of wheels
for projects that don't/won't/can't produce their own. But that's a
lot of work - the conda folks, and people like Christoph Gohlke, know
how much. Maybe by making the process simple enough (as described
above) might encourage someone to put together a wheel service along
the lines of Christoph's (but for all platforms, and in a format that
pip can access directly). Once something like that starts up, it may
attract additional volunteers, and grow from there. But it's still
quite a commitment from someone.

Paul