[Distutils] Plans for binary wheels, and PyPi and OS-X

[Donald Stufft]

On Oct 31, 2013, at 7:25 PM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth at gmail.com> wrote:
> I'm sure you could build your own broken Windows Python, but who
> bothers?
> 
> As long as we are clear that we are talking about a social difference here, not a technical one...
> 
> IMO it pretty much boils down to the fact that on Windows you
> are probably using the python.org version of Python and not linking
> with random shared libraries from C:\Program Files\, but on Linux you
> are probably using one of a few dozen distro x distro-release Pythons
> AND your extension probably dynamically links to some other things
> your specific distro provides AND maybe you are depending on some
> versioned symbols in glibc oh the horror.
> 
> On OS X I would hope you are uploading only wheels built with
> python.org-Python but maybe you aren't, everyone has their
> preferences.
> 
> yes, they do -- but what is the target audience here? yes, a lot of folks use macports, homebrew etc, fewer, but I'm sure some, build their own Python from scratch -- but these are NOT the people that we want binary wheels for -- they don't want them anyway.
> 
> The folks that we want to provide binary wheels for are NOT going to be building their own esoteric python, really, they are not.
> 
> The MacPython community has a long standing tradition of building binaries (if at all) that are compatible with the python.org builds (and secondarily with the Apple-supplied Python) -- that is what I'd like to see supported by PyPi -- just like Windows....
> 
> Sure, someone could upload some oddly-built binary wheel to PyPi -- then it would not work for most users, and they would get complaints and hopefully fix it -- just like uploading a package with any other kind of bug in it.
> 
> It is kind of a pain to build a truly portable binary package (when it depends on third-party compiled libs), but there is a small but committed group of folks doing that already -- let's make it easier to get stuff out there.
> 
> > Will a C extension built with Homebrew Python work with the Python Mac
> > OS X installer from python.org? Probably, but given how painful ABI
> > mismatches with shared libraries can be to debug, "probably" doesn't
> > cut it until someone has the chance to thoroughly review the potential
> > for problems.
> 
> I disagree:
> 
> 1) I don't care if homebrew built extensions work with other pythons -- you want to build with homebrew, create a homebrew recipe. -- there should be a policy about how binary packages posted on PyPi should be built.

Well it was more of we didn’t know, so we played it safe. I don’t personally have a Python.org Installer Python (mine come from Homebrew) and I know others use the System Python. My knowledge of compiled stuff is pretty slim tbh :)

> 
> 2) We're never going to find out what the problems are until we give it a try.
> 
> Fundamentally, I disagree with the premise here: "If we can't guarantee that anything anyone uploads will work for everyone, we shouldn't allow it" -- that's an unattainable goal.

It was less about guarantee and more about “Can we be reasonably sure this is going to work for most people”.

> 
> If we do want a more fool-proof approach, then the name auto-generated by wheel should include something that means "python.org-build" only if built with the python.org build.
> 
> And I suppose we could try to put check code in there to make sure that extensions aren't linked to outside libs. Actually, that would be a handy utility to have, even if it didn't enforce anything. (and by the way, it's rea;lly easy to build a binary for Windows that's linked to an external dll also -- we expect package builders to be careful with that...)
> 
> 
> I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX.
> 
> Sorry we weren't out there answering!

It’s not your fault! We realized this right before the pip release and sort of quickly added it after a quick check to see if anyone could tell us one way or another.

> 
> Linux is a different story -- not only are there a lot of variations out there, but there also is no obvious "standard" one could point to that we'd expect folks to build binary wheels for.
> 
> OS-X has (to much) variety  though it is less than Linux, and more to the point, there is a standard Python out there -- the python.org builds. And there is a tradition of building binaries for that standard. AFAIK, it is pretty much the ONLY build of Python that package maintainers support with binaries (if they support anything). 
> 
> > If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew?
> 
> probably not, but again, I don't care -- that's not what binary wheels on Python would be for. And more to the point -- this is a policy question -- don't upload a binary wheel to pypi that depends on homebrew (or anything else that Apple didn't provide)
> 
> Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them.-Chris
> 
> One thing I'm not clear on -- if you do :
> 
> pip install something
> 
> will pip preferentially select a binary wheel (if enabled on pypi?) -- that may be an issue as folks will surely try to pip install stuff with homebrew, macports, etc. pythons (though the wheels are more likely to work in that direction.

Right this second it would require opting in via —use-wheel to get wheel installs from a simple ``pip install something`` but there’s discussion around making wheels enabled by default.

> 
> -Chris




-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/1040d91b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/1040d91b/attachment.sig>