On 26 May 2016 at 19:20, Matthew Brett
I just wanted to make sure that we didn't lose out on starting a discussion of this problem.
The problem is of course caused by the runaway success of the wheel format, and I'm sure it can be solved in a sensible way, but however expressed, it's true that wheels have become so standard that we do need to think about automation for build and release, if we aren't going run into trouble. By trouble, I mean that users will often hit the situation where they don't get wheels when they expect to, and get turned off pypi / wheels as a result. I have personally put a great deal of work into building and releasing wheels, so that is something I'd really like to avoid.
So - can I humbly ask - what is the best way to get that discussion going?
I would certainly like to promote projects using existing CI systems for building wheels. It seems like a pretty simple approach that would cover a decent proportion of the problem. There's already a section in the Packaging User Guide explaining how to use Appveyor to build Windows binary wheels. If you don't have dependencies to deal with (e.g., you're simply providing a C speedup module) it's pretty simple. If you *do* have dependencies to deal with, I suspect you could arrange for them to be uploaded when you build somehow without too much issue - but I don't have projects like that so I can't really add that to the document. The biggest problem with Appveyor is that it's less well known than Travis. However, I don't know of many (frankly, any!) projects using this approach - whether that's because of a lack of awareness, or because it's not as useful as it seemed to me when I wrote it, or some other reason, I don't honestly know. Maybe we could get some feedback on what can be done to improve that section? Is there going to be any sort of packaging discussions at PyCon? Maybe it's a question that could be asked? (I'm not going to be there myself, unfortunately). For Linux/Unix, the problem was always the plethora of binary formats. If we can assume that manylinux builds will constitute a 90% solution here, maybe someone could contribute a section explaining how to add the building of manylinux wheels into a project's Travis config. I know almost nothing about OSX, but doesn't Travis offer OSX builders? Could they be used to build extensions in the same way as the above? Going beyond this level (which is basically making sure projects know they have easy access to build environments for the key platforms, and encouraging them to use them) is when it gets harder. Integrating artifacts from travis/appveyor into a release process would need some work, for example. Maybe a way forward here would be for some projects to try doing so, and writing up a set of "how we did it" notes that could over time turn into a set of recommended practices in the PUG. Or an interested 3rd party could probably do most of the experimentation and development, simply by forking a few typical projects and seeing what was needed. With all of the above, we'd end up with some (hopefully!) pretty good documentation of how to (relatively) painlessly add building of wheels for all major platforms into *any* project (that didn't have specialised issues to deal with). I'm not sure where we'd go from there. The next step seems to be to actually provide some sort of build service, or curated set of wheels for projects that don't/won't/can't produce their own. But that's a lot of work - the conda folks, and people like Christoph Gohlke, know how much. Maybe by making the process simple enough (as described above) might encourage someone to put together a wheel service along the lines of Christoph's (but for all platforms, and in a format that pip can access directly). Once something like that starts up, it may attract additional volunteers, and grow from there. But it's still quite a commitment from someone. Paul