[Distutils] Plans for binary wheels, and PyPi and OS-X

[Chris Barker]

On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth at gmail.com> wrote:

> I'm sure you could build your own broken Windows Python, but who
>  bothers?


As long as we are clear that we are talking about a social difference here,
not a technical one...

IMO it pretty much boils down to the fact that on Windows you
> are probably using the python.org version of Python and not linking
> with random shared libraries from C:\Program Files\, but on Linux you
> are probably using one of a few dozen distro x distro-release Pythons
> AND your extension probably dynamically links to some other things
> your specific distro provides AND maybe you are depending on some
> versioned symbols in glibc oh the horror.
>
> On OS X I would hope you are uploading only wheels built with
> python.org-Python but maybe you aren't, everyone has their
> preferences.
>

yes, they do -- but what is the target audience here? yes, a lot of folks
use macports, homebrew etc, fewer, but I'm sure some, build their own
Python from scratch -- but these are NOT the people that we want binary
wheels for -- they don't want them anyway.

The folks that we want to provide binary wheels for are NOT going to be
building their own esoteric python, really, they are not.

The MacPython community has a long standing tradition of building binaries
(if at all) that are compatible with the python.org builds (and secondarily
with the Apple-supplied Python) -- that is what I'd like to see supported
by PyPi -- just like Windows....

Sure, someone could upload some oddly-built binary wheel to PyPi -- then it
would not work for most users, and they would get complaints and hopefully
fix it -- just like uploading a package with any other kind of bug in it.

It is kind of a pain to build a truly portable binary package (when it
depends on third-party compiled libs), but there is a small
but committed group of folks doing that already -- let's make it easier to
get stuff out there.

> Will a C extension built with Homebrew Python work with the Python Mac
> > OS X installer from python.org? Probably, but given how painful ABI
> > mismatches with shared libraries can be to debug, "probably" doesn't
> > cut it until someone has the chance to thoroughly review the potential
> > for problems.
>

I disagree:

1) I don't care if homebrew built extensions work with other pythons -- you
want to build with homebrew, create a homebrew recipe. -- there should be a
policy about how binary packages posted on PyPi should be built.

2) We're never going to find out what the problems are until we give it a
try.

Fundamentally, I disagree with the premise here: "If we
can't guarantee that anything anyone uploads will work for everyone, we
shouldn't allow it" -- that's an unattainable goal.

If we do want a more fool-proof approach, then the name auto-generated by
wheel should include something that means "python.org-build" only if built
with the python.org build.

And I suppose we could try to put check code in there to make sure that
extensions aren't linked to outside libs. Actually, that would be a handy
utility to have, even if it didn't enforce anything. (and by the way, it's
rea;lly easy to build a binary for Windows that's linked to an external dll
also -- we expect package builders to be careful with that...)


I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut
> off Linux Wheels, at the same time I asked about Windows and OSX wheels,
> the answers I got from people were they would generally just work on
> Windows, and nobody gave me a straight answer on OSX.
>

Sorry we weren't out there answering!

Linux is a different story -- not only are there a lot of variations out
there, but there also is no obvious "standard" one could point to that we'd
expect folks to build binary wheels for.

OS-X has (to much) variety  though it is less than Linux, and more to the
point, there is a standard Python out there -- the python.org builds. And
there is a tradition of building binaries for that standard. AFAIK, it is
pretty much the ONLY build of Python that package maintainers support with
binaries (if they support anything).

> If you build a Wheel with Homebrew Python will it work on the official
> OSX installers? What if I have a library installed from Homebrew?


probably not, but again, I don't care -- that's not what binary wheels on
Python would be for. And more to the point -- this is a policy question --
don't upload a binary wheel to pypi that depends on homebrew (or anything
else that Apple didn't provide)

Essentially trying to figure out how likely it is that with the existing
> tags a wheel is going to work if we select based on them.-Chris
>

One thing I'm not clear on -- if you do :

pip install something

will pip preferentially select a binary wheel (if enabled on pypi?) -- that
may be an issue as folks will surely try to pip install stuff with
homebrew, macports, etc. pythons (though the wheels are more likely to work
in that direction.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131031/4aaefbdc/attachment-0001.html>